Data Extraction, Help Needed

[PXu]

Hi everyone,

I am in the process of starting a website/fastlane model business. However, one of my first steps is to extract a lot of data (No personal information) from another website. I was wondering how I can do so without it being illegal. Along with this, the best way to do so is if the website I get data from does not know at all, but if I do have to make a partnership/mutual agreement with them, how can I find ways to leverage myself to create enough value to strike a fair deal without them stealing my idea?

Thanks to anyone who has any advice for me. I greatly appreciate anyone who gives this question a shot, it is truly a huge roadblock for me at this point in my process.

 

Dislike ads? Remove them and support the forum: Subscribe to Fastlane Insiders.

[rwhyan]

Scraping + proxies

It's not illegal, but most sites disallow in TOS.

Depends if this is a one-time data extraction, or something you'll need to do repeatedly.

If the latter,

 

[PXu]

Scraping + proxies

It's not illegal, but most sites disallow in TOS.

Depends if this is a one-time data extraction, or something you'll need to do repeatedly.

If the latter,

I appreciate the insight. I would need to extract a lot of data to start out, but then once my website is up and running, I can look into partnerships once I have value/leverage. By using scraping and proxies, can I get a lot of data within a few weeks? I am trying to not get sued down the road if the website knows I got my data from them without asking. Of course, as stated, none of this would be personal information, or could hint at any personal information. Thanks again, God Bless.

P.s. if it is not illegal, what are the repercussions of such data extraction? I know this can be viewed as a "gray" area but I do have good intentions with the data

 

[SEBASTlAN]

There's a lot of scraping software out there, otherwise you can hire people on Upwork to do it for you. It's not illegal.

 

Dislike ads? Remove them and support the forum: Subscribe to Fastlane Insiders.

[svekk1]

It's how you use the data that you scraped that might be illegal rather than the scraping itself. If the data is publicly available to anyone it should'nt cause issues. If it's behind a paywall or login or something it's a more delicate question.

You can also check what the site allows by adding "robots.txt" after the url. This is more from an ethical point of view than a legal one.

"www.examplesite.whatever/robots.txt"

 

[PXu]

There's a lot of scraping software out there, otherwise you can hire people on Upwork to do it for you. It's not illegal.

Thanks for the information. If I did that, I would have to get them to sign NDA's.

 

[PXu]

It's how you use the data that you scraped that might be illegal rather than the scraping itself. If the data is publicly available to anyone it should'nt cause issues. If it's behind a paywall or login or something it's a more delicate question.

You can also check what the site allows by adding "robots.txt" after the url. This is more from an ethical point of view than a legal one.

"www.examplesite.whatever/robots.txt"

The data is behind a paywall. I would have to subscribe to the website for a fixed amount a year to gain the data. Along with that, I would use the data by analyzing it and creating key trends and putting it out on my own website. Is this something that sounds like it can be illegal? I do have good intentions in helping people.

 

Dislike ads? Remove them and support the forum: Subscribe to Fastlane Insiders.

[svekk1]

You would still need their consent in case you want a non-profit or charity. And if they consent they would probably have a more efficient way to provide data for you.

In the other case, try to find out if they act as a data provider, so you can buy data from them.
If no, tryto figure out where they get the data. Do they generate it themself? Or simply collect it? Or do they have a data provider? In the latter case you can buy the data from their provider.

 

[PXu]

You would still need their consent in case you want a non-profit or charity. And if they consent they would probably have a more efficient way to provide data for you.

In the other case, try to find out if they act as a data provider, so you can buy data from them.
If no, tryto figure out where they get the data. Do they generate it themself? Or simply collect it? Or do they have a data provider? In the latter case you can buy the data from their provider.

God bless you. Thank you for the info. If I buy data from them, they would need to know what I am doing with their data for ethical purposes. What do I tell them if I do not want to tell them my business idea?

 

[Happyheart]

God bless you. Thank you for the info. If I buy data from them, they would need to know what I am doing with their data for ethical purposes. What do I tell them if I do not want to tell them my business idea?

Why not tell them something generic? That you are looking to market to xy population for project z. It would be more honest and if you are not exactly the competition, then they will tell you their price.
On the other hand, if you try stealing their data it may give you a lot of stress and may provide problems. I have had people asking for information and then putting it directly on their website. That doesn‘t make me want to work with them in the future.

 

Dislike ads? Remove them and support the forum: Subscribe to Fastlane Insiders.

[PXu]

Why not tell them something generic? That you are looking to market to xy population for project z. It would be more honest and if you are not exactly the competition, then they will tell you their price.
On the other hand, if you try stealing their data it may give you a lot of stress and may provide problems. I have had people asking for information and then putting it directly on their website. That doesn‘t make me want to work with them in the future.

Why not tell them something generic? That you are looking to market to xy population for project z. It would be more honest and if you are not exactly the competition, then they will tell you their price.
On the other hand, if you try stealing their data it may give you a lot of stress and may provide problems. I have had people asking for information and then putting it directly on their website. That doesn‘t make me want to work with them in the future.

If I go down the route of trying to partner with them. That is what I would tell them. However, to touch on your second point, I would be taking data from their website, visualizing it through easy to understand graphs/charts, and the putting it onto my website, all the while targeting a different market than the website I pull data from. I do not know if they would be interested in partnering with me if I am doing that.

 

[svekk1]

If they are into selling data to you, they will not care until you aren't a direct competitor.

 

[PXu]

If they are into selling data to you, they will not care until you aren't a direct competitor.

Thanks for the advice. I was thinking the same thing. Just buy the monthly subscription to access the data, extract it using a data scraper, use the data to do what i am planning to do with it and then make my website/business around it. They will not know/care where I got my data from until I am big enough to compete with them, but by that time, I will be able to source my own data. Does that sound ethical?

 

Dislike ads? Remove them and support the forum: Subscribe to Fastlane Insiders.

[svekk1]

Thanks for the advice. I was thinking the same thing. Just buy the monthly subscription to access the data, extract it using a data scraper, use the data to do what i am planning to do with it and then make my website/business around it. They will not know/care where I got my data from until I am big enough to compete with them, but by that time, I will be able to source my own data. Does that sound ethical?

I think we are back at square one.

In my experience, if you are using their data to make your own product, you cannot aproach it this way. You have to buy the data from them (accessing their platform is not equal to buying their data and then using it for you own profit)

You'll have to find out if they are willing to sell data trough an API or flat files to you.

Do you know if they source the data themselfs or use a data provider?

It might be worth doing a web search to see if there is a data provider in that field, whose business model is to provide the data to companies that need it rather than operating a platform.

For example for financial data Refinitiv is a data provider

 

[eramart]

Legal matters aside, if you scrape data using your account, they will probably catch and ban you

 

[UK_Mike]

I expect that the terms and conditions for signing up for the paid account will confirm what you can do with their data. "Illegal" is a broad term, and will probably vary depending on which country/countries you and the web site owner are in, but even if it's not an actual crime, I'd expect them to take some action if they find out you're doing it. The fact that it's behind a paywall suggests that there is some value to the data, so they're unlikely to take your theft of it without any action.

Once your site is done, how will you feel if someone scrapes all your data and uses it for something else?

 

Dislike ads? Remove them and support the forum: Subscribe to Fastlane Insiders.

[PXu]

I think we are back at square one.

In my experience, if you are using their data to make your own product, you cannot aproach it this way. You have to buy the data from them (accessing their platform is not equal to buying their data and then using it for you own profit)

You'll have to find out if they are willing to sell data trough an API or flat files to you.

Do you know if they source the data themselfs or use a data provider?

It might be worth doing a web search to see if there is a data provider in that field, whose business model is to provide the data to companies that need it rather than operating a platform.

For example for financial data Refinitiv is a data provider

Thanks for the further clarification. I will do some research into this today. I do not know whether they buy their data or if they source it themselves.

 

[PXu]

Legal matters aside, if you scrape data using your account, they will probably catch and ban you

Appreciate the comment. Agreed. I do not want to do anything unethical or borderline illegal. I am going to meet with a corporate attorney soon to discuss how I can go about tackling this problem

 

[PXu]

I expect that the terms and conditions for signing up for the paid account will confirm what you can do with their data. "Illegal" is a broad term, and will probably vary depending on which country/countries you and the web site owner are in, but even if it's not an actual crime, I'd expect them to take some action if they find out you're doing it. The fact that it's behind a paywall suggests that there is some value to the data, so they're unlikely to take your theft of it without any action.

Once your site is done, how will you feel if someone scrapes all your data and uses it for something else?

Thanks for the comment Mike. I did take a look at their ToS and EULA, but neither of them state if I can extract their data or not. From my understanding, if it goes unsaid, I should go about it like it is prohibited.

To address your last point, I would also be hesitant if someone were to scrape my data, but on the other hand, as long as I am not trying to directly compete with them, I do not see a big problem with it.

 

Dislike ads? Remove them and support the forum: Subscribe to Fastlane Insiders.

[UK_Mike]

To address your last point, I would also be hesitant if someone were to scrape my data, but on the other hand, as long as I am not trying to directly compete with them, I do not see a big problem with it.

I guess you also have to consider what it costs them to gather and maintain that data set - whether they gather it themselves or license it from a data provider, it will be costing them. I don't know what the data is made up of - if it's personal details, for example, they may have agreed with each individual that they will not share the data with a third party.

 

[PXu]

I guess you also have to consider what it costs them to gather and maintain that data set - whether they gather it themselves or license it from a data provider, it will be costing them. I don't know what the data is made up of - if it's personal details, for example, they may have agreed with each individual that they will not share the data with a third party.

It is not personal data. None of the data I need to collect would directly or indirectly link to anyone’s personal data. I just have to find the right angle to leverage a mutually beneficial partnership while I do not have much to offer while they have all the power.