Social network aggregation

C

Clifford Leong

Member
Joined
Sep 27, 2011
Messages
170
Reaction score
0
Anyone familiar with Social network aggregation?

Would it be possible to integrate this on a website and filter only related post?

Eg. Any post related to or contains "Soccernet" on Facebook, Twitter, instagram etc be automatically shared/posted on my site?

http://stackla.com/
An example of what i need although certain things needs to be changed.

Please advise.
 
Last edited:
S

stupidbodo

Junior Member
Joined
Mar 26, 2010
Messages
89
Reaction score
0
it might seem trivial but I reckon you need a couple of things planned out nicely in order to get this out. Here are the things to consider.

Source
You need to plan where do you want to grab the data and news from.

Ingesting/Grabbing Data
You need to consider do the sources have a direct api for you to access the data? Or do you need to run a scraper to extract the data? I think this component is tricky because every sources will have different methodology for you to extract their data so you have to consider carefully. Also, how is your data going into your database? Is it via a message queue? What kind of database are you going to use?

Cleaning/Filtering Data
You need to consider things like nude/spammy or whatever stuffs there could be. There might also be chances that there will be duplicate posts since stuffs like retweet happens quite often.

Updating
You need an easy way to update your extracting/scraping methods(possibly having a UI on your admin side) because sites change/update all the time. There can also be a possibility that the site will block your scraper's ip address.

Admin UI
Followup from last point, we are not talking about what your users see. Your admin needs an easy way to CRUD, start/pause the extracting rules, scraping methods. It's better if you have an admin UI rather than changing directly from source or giving shell commands.

Infrastructure
This is probably an important one. You need a way to know or monitor how fast your scrapers are running. What if your scraper goes down? What are you going to do if it goes down? Is there going to be an auto-healing system? Some best things to remember is to "try to automate as much as you can" and "monitor everything because everything can fail".


That's about the hindsight of it. We are not even talking about the frontend UI that your user sees and I can go more specific but these are the things you need to consider if you want to get started. Surprise by the complexity :eek:? Yeah that's the kind of stuffs you need to consider when you grow out of the normal mom and pop site you see and that's what many people who thinks that been an internet startup is easy don't see.
 
C

Clifford Leong

Member
Joined
Sep 27, 2011
Messages
170
Reaction score
0
Thanks stupidbodo for your time.

I am just checking how feasible it would be to do so. i was'nt planning to do it on my own. :)

Judging from your reply, i should speak to a developer and talk about the factors you outlined.
 
davidktw

davidktw

Arch-Supremacy Member
Joined
Apr 15, 2010
Messages
13,547
Reaction score
1,301
stupidbodo said:
it might seem trivial but I reckon you need a couple of things planned out nicely in order to get this out. Here are the things to consider.

Source
You need to plan where do you want to grab the data and news from.

Ingesting/Grabbing Data
You need to consider do the sources have a direct api for you to access the data? Or do you need to run a scraper to extract the data? I think this component is tricky because every sources will have different methodology for you to extract their data so you have to consider carefully. Also, how is your data going into your database? Is it via a message queue? What kind of database are you going to use?

Cleaning/Filtering Data
You need to consider things like nude/spammy or whatever stuffs there could be. There might also be chances that there will be duplicate posts since stuffs like retweet happens quite often.

Updating
You need an easy way to update your extracting/scraping methods(possibly having a UI on your admin side) because sites change/update all the time. There can also be a possibility that the site will block your scraper's ip address.

Admin UI
Followup from last point, we are not talking about what your users see. Your admin needs an easy way to CRUD, start/pause the extracting rules, scraping methods. It's better if you have an admin UI rather than changing directly from source or giving shell commands.

Infrastructure
This is probably an important one. You need a way to know or monitor how fast your scrapers are running. What if your scraper goes down? What are you going to do if it goes down? Is there going to be an auto-healing system? Some best things to remember is to "try to automate as much as you can" and "monitor everything because everything can fail".


That's about the hindsight of it. We are not even talking about the frontend UI that your user sees and I can go more specific but these are the things you need to consider if you want to get started. Surprise by the complexity :eek:? Yeah that's the kind of stuffs you need to consider when you grow out of the normal mom and pop site you see and that's what many people who thinks that been an internet startup is easy don't see.
Click to expand...

Your consideration for the social network integration is thorough, however, it's not as daunting as it seems (looking from the large passage) :)

I have done it before with Twitter, Facebook and Instagram. The API to scrap based on hashtags or keywords are found available in these social network services.

Without using any SDK, pure RESTful approach to query from these API are in my opinion very easy to do. You can even do it through CURL if you don't mind the slow in performance :) If not, just a simple running daemon will do just fine.

Extracting of information with these services are using purely JSON and hence very easy to consume. If you are using MongoDB, it's even better since it can be directly pushed into the database immediately without data massaging.

Retweets are easy to identify since they are provided in Twitter JSON response. Duplicated tweets are easily filtered out using the unique id given in each tweets, same goes for Instagram and Facebook too.

As in all production level software, High Availability is a consideration but not difficult to achieve. In the past, we use to always have at least a pair of servers to do this. But if the introduction of cloud computing, using AWS as an example. I propose the use of autoscaling of at most 1 instance to perform the HA for these scraper. The reason is all these 3 social networks mainly uses polling approach to get the feeds. The continuity is using the object id or unique id as a reference upon each poll, hence there is no need for 100% uptime since the next poll will pick up from where it left off.

As for performance, there is no need to be too concern since performance on your side will most likely be throttle by the social network service instead. There is only so fast you can get from these services.

As for administrative console for nude and spamming, it's up to the requirement of the system, generally they are not of great concern to most services unless the requirement of profanity and obscenity filtering is required.

As for the UI, surely it's something not rocket science. While it may not look as nice and polished as the example given, surely a decent display that is refreshed upon page load is possible. How fluid and presentable it is will solely depend on the skill set, effort and cost invested.
 
davidktw

davidktw

Arch-Supremacy Member
Joined
Apr 15, 2010
Messages
13,547
Reaction score
1,301
Clifford Leong said:
Thanks stupidbodo for your time.

I am just checking how feasible it would be to do so. i was'nt planning to do it on my own. :)

Judging from your reply, i should speak to a developer and talk about the factors you outlined.
Click to expand...

Yes please to discuss with the developer on how it should be executed. My last experience working with all these services are using purely perl only for feeds scrapping, nothing fanciful, but is currently in production with one of the customer. It wouldn't be nice for me to reveal, but I can assure you it's a high profile customer :)

Please refer to my response in the above for some details on some of the implementation that are highlighted by stupidbobo.
 
Last edited:
davidktw

davidktw

Arch-Supremacy Member
Joined
Apr 15, 2010
Messages
13,547
Reaction score
1,301
Clifford Leong said:
Hi Davidktw,

Thanks for your reply, judging from your post.. seems easier ah?
Click to expand...

Whether it's easy or not will depends on how much you require.
There are some learning curve to integration with these social networks.
Common knowledge are Open Authentication, checksum calculation for OAuth 1.0, good understanding of HTTP/S request/response model etc.

Each social network has its own set of intrinsic workflow despite they may belong to the same authentication and access model. Also from time to time, their access and security model changes, making it hard for beginners since there will be numerous versions of online advices and articles but only the latest one matches the most recent implementation. These social networks system also have running targets for their implement and the lifecycle of their implementation may only last for like 1 year before they move on to another slightly different model. That's the confusion I get when I first started.

Just for twitter alone, I give you a premier for how to query for hashtags to your relevance.

Lets say I'm interested in EPL, so the hashtag I'm interested in is #epl.

First I must have a Twitter account and also a twitter application created. 2 important information is the consumer key and secret. This is a concept within OAuth. Previously for Twitter search API 1.0, there is no need for authentication, but it's no longer true for API 1.1. Using the consumer key and secret, you base64 encode them using the following technique
Code: 
base64("<CONSUMER_KEY>:<CONSUMER_SECRET>")

If I will to use the unix shell commands
Code: 
echo -n 'v8PCZhSHGuFnzXQXAAAmVg:915pmuaHq4hxbpuo6i8ZP7kafvwOAgXXXyd2KQU6Y' | base64
djhQQ1poU0hHdUZuelhRWEFBQW1WZzo5MTVwbXVhSHE0aHhicHVvNmk4WlA3a2FmdndPQWdYWFh5ZDJLUVU2WQ==

Then I need to get a bearer token from Twitter as such
Code: 
curl --form 'grant_type=client_credentials' -H 'Authorization: Basic djhQQ1poU0hHdUZuelhRWEFBQW1WZzo5MTVwbXVhSHE0aHhicHVvNmk4WlA3a2FmdndPQ' 'https://api.twitter.com/oauth2/token'
{"access_token":"AAAAAAAAAAAAAAAAAAAAALXxTBBBBBBBne83fN5bOk7spUigV308o5B23r4%3DVjKeREmpjsXGPCVgQQ1zcobkM5cOY2E0tm7Hgsgt9O4","token_type":"bearer"}

Using the returned bearer token, I can use the following curl command to obtain a json response for all tweets recently with hashtag "EPL"
Code: 
curl -H 'Authorization: Bearer AAAAAAAAAAAAAAAAAAAAALXxTBBBBBBBne83fN5bOk7spUigV308o5B23r4%3DVjKeREmpjsXGPCVgQQ1zcobkM5cOY2E0tm7Hgsgt9O4' 'https://api.twitter.com/1.1/search/tweets.json?q=epl'

All the above token presented here are masked, so you can't use them again. I'm just illustrating you the steps for Twitter Search API 1.1

Below is the actual json response from Twitter, after truncating a good number of tweets with only the first 2 left. It is still a valid output from Twitter.

How you want to present these feeds will be up to your imagination. It can be as complication as the example site you have given, or simply just a top-down single field list on the webpage.

Perform all these steps inside any programming languages will get you the same thing. I generally use Perl because it is one of the most powerful programming language for text processing and I'm pretty fluent in it. You can use any other languages as you wish.

  • Is there any learning curve? Yes.
  • Is it really that hard? I don't think so since there are so many sites together implementing features with feeds from all these social networks.
  • Do you need planning on how to architect a solution that can achieve what you want in a production environment? Definitely.
Code: 
{
    "search_metadata": {
        "completed_in": 0.027, 
        "count": 15, 
        "max_id": 375666998694658048, 
        "max_id_str": "375666998694658048", 
        "next_results": "?max_id=375666721266606079&q=epl&include_entities=1", 
        "query": "epl", 
        "refresh_url": "?since_id=375666998694658048&q=epl&include_entities=1", 
        "since_id": 0, 
        "since_id_str": "0"
    }, 
    "statuses": [
        {
            "contributors": null, 
            "coordinates": null, 
            "created_at": "Thu Sep 05 17:09:26 +0000 2013", 
            "entities": {
                "hashtags": [], 
                "symbols": [], 
                "urls": [], 
                "user_mentions": [
                    {
                        "id": 191515754, 
                        "id_str": "191515754", 
                        "indices": [
                            3, 
                            15
                        ], 
                        "name": "Mark Pitman", 
                        "screen_name": "markpitman1"
                    }
                ]
            }, 
            "favorite_count": 0, 
            "favorited": false, 
            "geo": null, 
            "id": 375666998694658048, 
            "id_str": "375666998694658048", 
            "in_reply_to_screen_name": null, 
            "in_reply_to_status_id": null, 
            "in_reply_to_status_id_str": null, 
            "in_reply_to_user_id": null, 
            "in_reply_to_user_id_str": null, 
            "lang": "en", 
            "metadata": {
                "iso_language_code": "en", 
                "result_type": "recent"
            }, 
            "place": null, 
            "retweet_count": 28, 
            "retweeted": false, 
            "retweeted_status": {
                "contributors": null, 
                "coordinates": null, 
                "created_at": "Thu Sep 05 17:02:24 +0000 2013", 
                "entities": {
                    "hashtags": [], 
                    "symbols": [], 
                    "urls": [], 
                    "user_mentions": []
                }, 
                "favorite_count": 4, 
                "favorited": false, 
                "geo": null, 
                "id": 375665228777660416, 
                "id_str": "375665228777660416", 
                "in_reply_to_screen_name": null, 
                "in_reply_to_status_id": null, 
                "in_reply_to_status_id_str": null, 
                "in_reply_to_user_id": null, 
                "in_reply_to_user_id_str": null, 
                "lang": "en", 
                "metadata": {
                    "iso_language_code": "en", 
                    "result_type": "recent"
                }, 
                "place": null, 
                "retweet_count": 28, 
                "retweeted": false, 
                "source": "web", 
                "text": "Valencia charging \u00a37 a ticket for their Europa League match against Swansea City. You would struggle to feed yourself for that in the EPL.", 
                "truncated": false, 
                "user": {
                    "contributors_enabled": false, 
                    "created_at": "Thu Sep 16 17:02:37 +0000 2010", 
                    "default_profile": false, 
                    "default_profile_image": false, 
                    "description": "Welsh football correspondent and blogger. Regular contributor to a number of websites including http://t.co/dWvvFD8Dve and WalesOnline. Also @UEFAcomMPitman.", 
                    "entities": {
                        "description": {
                            "urls": [
                                {
                                    "display_url": "UEFA.com", 
                                    "expanded_url": "http://UEFA.com", 
                                    "indices": [
                                        96, 
                                        118
                                    ], 
                                    "url": "http://t.co/dWvvFD8Dve"
                                }
                            ]
                        }, 
                        "url": {
                            "urls": [
                                {
                                    "display_url": "markpitman1.com", 
                                    "expanded_url": "http://www.markpitman1.com", 
                                    "indices": [
                                        0, 
                                        22
                                    ], 
                                    "url": "http://t.co/yZeX9cszY2"
                                }
                            ]
                        }
                    }, 
                    "favourites_count": 0, 
                    "follow_request_sent": null, 
                    "followers_count": 2502, 
                    "following": null, 
                    "friends_count": 956, 
                    "geo_enabled": false, 
                    "id": 191515754, 
                    "id_str": "191515754", 
                    "is_translator": false, 
                    "lang": "en", 
                    "listed_count": 73, 
                    "location": "Port Talbot, Wales", 
                    "name": "Mark Pitman", 
                    "notifications": null, 
                    "profile_background_color": "9AE4E8", 
                    "profile_background_image_url": "http://a0.twimg.com/profile_background_images/744781811/b93246d6929c1eb44267ea1f900b0f94.jpeg", 
                    "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/744781811/b93246d6929c1eb44267ea1f900b0f94.jpeg", 
                    "profile_background_tile": false, 
                    "profile_banner_url": "https://pbs.twimg.com/profile_banners/191515754/1348146947", 
                    "profile_image_url": "http://a0.twimg.com/profile_images/378800000147616389/87f9a38f7a81e800f1cfb52cb6bfe109_normal.jpeg", 
                    "profile_image_url_https": "https://si0.twimg.com/profile_images/378800000147616389/87f9a38f7a81e800f1cfb52cb6bfe109_normal.jpeg", 
                    "profile_link_color": "000DFF", 
                    "profile_sidebar_border_color": "000000", 
                    "profile_sidebar_fill_color": "D6FFF8", 
                    "profile_text_color": "000000", 
                    "profile_use_background_image": true, 
                    "protected": false, 
                    "screen_name": "markpitman1", 
                    "statuses_count": 18138, 
                    "time_zone": "London", 
                    "url": "http://t.co/yZeX9cszY2", 
                    "utc_offset": 3600, 
                    "verified": false
                }
            }, 
            "source": "<a href=\"http://twitter.com/download/android\" rel=\"nofollow\">Twitter for Android</a>", 
            "text": "RT @markpitman1: Valencia charging \u00a37 a ticket for their Europa League match against Swansea City. You would struggle to feed yourself for \u2026", 
            "truncated": false, 
            "user": {
                "contributors_enabled": false, 
                "created_at": "Sat Jun 27 22:28:19 +0000 2009", 
                "default_profile": false, 
                "default_profile_image": false, 
                "description": "S'up!! I'm Mathew, 19. I love music, Football, Tennis, Youtube and gaming! :) I support the greatest team in football Cardiff City FC, ST holder.", 
                "entities": {
                    "description": {
                        "urls": []
                    }, 
                    "url": {
                        "urls": [
                            {
                                "display_url": "facebook.com/MathewL23", 
                                "expanded_url": "http://www.facebook.com/MathewL23", 
                                "indices": [
                                    0, 
                                    22
                                ], 
                                "url": "http://t.co/fX9kEGQjz5"
                            }
                        ]
                    }
                }, 
                "favourites_count": 495, 
                "follow_request_sent": null, 
                "followers_count": 1238, 
                "following": null, 
                "friends_count": 1721, 
                "geo_enabled": true, 
                "id": 51575884, 
                "id_str": "51575884", 
                "is_translator": false, 
                "lang": "en", 
                "listed_count": 5, 
                "location": "Wales, Cardiff", 
                "name": "Mathew Ling", 
                "notifications": null, 
                "profile_background_color": "FFFFFF", 
                "profile_background_image_url": "http://a0.twimg.com/profile_background_images/845652281/0ac9451d1b5d492e6ad705b3c225850a.jpeg", 
                "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/845652281/0ac9451d1b5d492e6ad705b3c225850a.jpeg", 
                "profile_background_tile": true, 
                "profile_banner_url": "https://pbs.twimg.com/profile_banners/51575884/1369663743", 
                "profile_image_url": "http://a0.twimg.com/profile_images/3569406328/689cda3f2c0c0727ecf35aef11f13d03_normal.jpeg", 
                "profile_image_url_https": "https://si0.twimg.com/profile_images/3569406328/689cda3f2c0c0727ecf35aef11f13d03_normal.jpeg", 
                "profile_link_color": "FF0000", 
                "profile_sidebar_border_color": "FFFFFF", 
                "profile_sidebar_fill_color": "C0DFEC", 
                "profile_text_color": "333333", 
                "profile_use_background_image": true, 
                "protected": false, 
                "screen_name": "MathewL23", 
                "statuses_count": 17093, 
                "time_zone": "London", 
                "url": "http://t.co/fX9kEGQjz5", 
                "utc_offset": 3600, 
                "verified": false
            }
        }, 
        {
            "contributors": null, 
            "coordinates": null, 
            "created_at": "Thu Sep 05 17:09:21 +0000 2013", 
            "entities": {
                "hashtags": [], 
                "symbols": [], 
                "urls": [], 
                "user_mentions": [
                    {
                        "id": 191515754, 
                        "id_str": "191515754", 
                        "indices": [
                            3, 
                            15
                        ], 
                        "name": "Mark Pitman", 
                        "screen_name": "markpitman1"
                    }
                ]
            }, 
            "favorite_count": 0, 
            "favorited": false, 
            "geo": null, 
            "id": 375666977458487296, 
            "id_str": "375666977458487296", 
            "in_reply_to_screen_name": null, 
            "in_reply_to_status_id": null, 
            "in_reply_to_status_id_str": null, 
            "in_reply_to_user_id": null, 
            "in_reply_to_user_id_str": null, 
            "lang": "en", 
            "metadata": {
                "iso_language_code": "en", 
                "result_type": "recent"
            }, 
            "place": null, 
            "retweet_count": 28, 
            "retweeted": false, 
            "retweeted_status": {
                "contributors": null, 
                "coordinates": null, 
                "created_at": "Thu Sep 05 17:02:24 +0000 2013", 
                "entities": {
                    "hashtags": [], 
                    "symbols": [], 
                    "urls": [], 
                    "user_mentions": []
                }, 
                "favorite_count": 4, 
                "favorited": false, 
                "geo": null, 
                "id": 375665228777660416, 
                "id_str": "375665228777660416", 
                "in_reply_to_screen_name": null, 
                "in_reply_to_status_id": null, 
                "in_reply_to_status_id_str": null, 
                "in_reply_to_user_id": null, 
                "in_reply_to_user_id_str": null, 
                "lang": "en", 
                "metadata": {
                    "iso_language_code": "en", 
                    "result_type": "recent"
                }, 
                "place": null, 
                "retweet_count": 28, 
                "retweeted": false, 
                "source": "web", 
                "text": "Valencia charging \u00a37 a ticket for their Europa League match against Swansea City. You would struggle to feed yourself for that in the EPL.", 
                "truncated": false, 
                "user": {
                    "contributors_enabled": false, 
                    "created_at": "Thu Sep 16 17:02:37 +0000 2010", 
                    "default_profile": false, 
                    "default_profile_image": false, 
                    "description": "Welsh football correspondent and blogger. Regular contributor to a number of websites including http://t.co/dWvvFD8Dve and WalesOnline. Also @UEFAcomMPitman.", 
                    "entities": {
                        "description": {
                            "urls": [
                                {
                                    "display_url": "UEFA.com", 
                                    "expanded_url": "http://UEFA.com", 
                                    "indices": [
                                        96, 
                                        118
                                    ], 
                                    "url": "http://t.co/dWvvFD8Dve"
                                }
                            ]
                        }, 
                        "url": {
                            "urls": [
                                {
                                    "display_url": "markpitman1.com", 
                                    "expanded_url": "http://www.markpitman1.com", 
                                    "indices": [
                                        0, 
                                        22
                                    ], 
                                    "url": "http://t.co/yZeX9cszY2"
                                }
                            ]
                        }
                    }, 
                    "favourites_count": 0, 
                    "follow_request_sent": null, 
                    "followers_count": 2502, 
                    "following": null, 
                    "friends_count": 956, 
                    "geo_enabled": false, 
                    "id": 191515754, 
                    "id_str": "191515754", 
                    "is_translator": false, 
                    "lang": "en", 
                    "listed_count": 73, 
                    "location": "Port Talbot, Wales", 
                    "name": "Mark Pitman", 
                    "notifications": null, 
                    "profile_background_color": "9AE4E8", 
                    "profile_background_image_url": "http://a0.twimg.com/profile_background_images/744781811/b93246d6929c1eb44267ea1f900b0f94.jpeg", 
                    "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/744781811/b93246d6929c1eb44267ea1f900b0f94.jpeg", 
                    "profile_background_tile": false, 
                    "profile_banner_url": "https://pbs.twimg.com/profile_banners/191515754/1348146947", 
                    "profile_image_url": "http://a0.twimg.com/profile_images/378800000147616389/87f9a38f7a81e800f1cfb52cb6bfe109_normal.jpeg", 
                    "profile_image_url_https": "https://si0.twimg.com/profile_images/378800000147616389/87f9a38f7a81e800f1cfb52cb6bfe109_normal.jpeg", 
                    "profile_link_color": "000DFF", 
                    "profile_sidebar_border_color": "000000", 
                    "profile_sidebar_fill_color": "D6FFF8", 
                    "profile_text_color": "000000", 
                    "profile_use_background_image": true, 
                    "protected": false, 
                    "screen_name": "markpitman1", 
                    "statuses_count": 18138, 
                    "time_zone": "London", 
                    "url": "http://t.co/yZeX9cszY2", 
                    "utc_offset": 3600, 
                    "verified": false
                }
            }, 
            "source": "web", 
            "text": "RT @markpitman1: Valencia charging \u00a37 a ticket for their Europa League match against Swansea City. You would struggle to feed yourself for \u2026", 
            "truncated": false, 
            "user": {
                "contributors_enabled": false, 
                "created_at": "Sun Jun 28 19:37:53 +0000 2009", 
                "default_profile": true, 
                "default_profile_image": false, 
                "description": "Beer, bets and football, with a little NFL thrown in. Nerd. #WFC", 
                "entities": {
                    "description": {
                        "urls": []
                    }
                }, 
                "favourites_count": 178, 
                "follow_request_sent": null, 
                "followers_count": 164, 
                "following": null, 
                "friends_count": 410, 
                "geo_enabled": true, 
                "id": 51818678, 
                "id_str": "51818678", 
                "is_translator": false, 
                "lang": "en", 
                "listed_count": 2, 
                "location": "Walsall", 
                "name": "Ben Fereday", 
                "notifications": null, 
                "profile_background_color": "C0DEED", 
                "profile_background_image_url": "http://a0.twimg.com/images/themes/theme1/bg.png", 
                "profile_background_image_url_https": "https://si0.twimg.com/images/themes/theme1/bg.png", 
                "profile_background_tile": false, 
                "profile_banner_url": "https://pbs.twimg.com/profile_banners/51818678/1374596919", 
                "profile_image_url": "http://a0.twimg.com/profile_images/378800000307189232/b4642c7cca55b25f63485a5858b370a4_normal.jpeg", 
                "profile_image_url_https": "https://si0.twimg.com/profile_images/378800000307189232/b4642c7cca55b25f63485a5858b370a4_normal.jpeg", 
                "profile_link_color": "0084B4", 
                "profile_sidebar_border_color": "C0DEED", 
                "profile_sidebar_fill_color": "DDEEF6", 
                "profile_text_color": "333333", 
                "profile_use_background_image": true, 
                "protected": false, 
                "screen_name": "latviancheese", 
                "statuses_count": 4631, 
                "time_zone": "Amsterdam", 
                "url": null, 
                "utc_offset": 7200, 
                "verified": false
            }
        }
    ]
}
 
Last edited:
C

Clifford Leong

Member
Joined
Sep 27, 2011
Messages
170
Reaction score
0
Hi Davidktw,

Thanks for your reply, i see you have put in alot of effort!

If i had a webpage, i can pull out post off other social media sites with the usernames and stuff as well?

What i intend to do would require their name and would be nice if all of it was in a standard format

Name
##Post##
Share / Like

So like you said if someone #hashtag #BPL on facebook/twitter/instagram

it would automatically be pulled onto my web displaying:

Person "A" #BPL
##Post##
Share / Like

Person "B" #BPL
##Post##
Share / Like

Person "C" #BPL
##Post##
Share / Like

If the word wasn't hashtagged, can i still actually pull it out?
 
davidktw

davidktw

Arch-Supremacy Member
Joined
Apr 15, 2010
Messages
13,547
Reaction score
1,301
Clifford Leong said:
Hi Davidktw,

Thanks for your reply, i see you have put in alot of effort!

If i had a webpage, i can pull out post off other social media sites with the usernames and stuff as well?

What i intend to do would require their name and would be nice if all of it was in a standard format

Name
##Post##
Share / Like

So like you said if someone #hashtag #BPL on facebook/twitter/instagram

it would automatically be pulled onto my web displaying:

Person "A" #BPL
##Post##
Share / Like

Person "B" #BPL
##Post##
Share / Like

Person "C" #BPL
##Post##
Share / Like

If the word wasn't hashtagged, can i still actually pull it out?
Click to expand...

What exactly you can get off the social networks will be dependent on their offerings, you are advised to check with the corresponding API to have a better understanding on the scope and limitations.

My personal experience is getting the message, the media shared, who shared it and when the event happens is there across all the services' feeds. Mainly the differences is the format of the feed and the verbosity of the feeds.

So far all feeds are using polling approach. Twitter does have persistent polling for like following other users event, but may not be applicable to your usage.

For Share/Like, this integration will require OAuth integration. If you are using Javascript SDK for Facebook, it's not difficult. If you are using Graph API (for Facebook), it's slightly more involving.

Your Share/Like feature may also stretch across multiple other providers. You can choose to use frontend javascript widgets found in here for e.g.: Top 10 Social Sharing Buttons for Your Website » Practical Ecommerce

If I remember correctly, Twitter does allow text search, not just hashtag, the others you may need to trial it.
 
You must log in or register to reply here.
Important Forum Advisory Note
This forum is moderated by volunteer moderators who will react only to members' feedback on posts. Moderators are not employees or representatives of HWZ Forums. Forum members and moderators are responsible for their own posts. Please refer to our Community Guidelines and Standards and Terms and Conditions for more information.
Top