What's new

Search results

  1. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    I just checked it out and they are not in there but I pretty much figured out the APIs using Chrome Dev tools by myself so I am good there. A couple of the websites share the same API but most do not and those are the comparisons I am talking about. But thanks for the suggestion!
  2. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    -What appears to be a trivial problem matching up similar data from different websites is actually quite complex. For example, it is hard for a computer to understand that AAPL is referring to Apple if you can't pre-generate lists that tell it that AAPL is Apple. I am working with different...
  3. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    -I have been working on the script to clean up the data but everything looks good.
  4. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    What I did up to now: -Spiders are straightened out and I found a proxy service and implemented other measures for preventing my spiders from getting banned -Going to work on cleaning up the data now then I should be good to go from a data standpoint, then on to the website
  5. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    7/6/2020 and 7/7/2020 -Successfully deployed spiders on scrapinghub. -Creating another script to compare all of the items and upload them to SQL database
  6. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    7/5/2020 -Cleaned up all spiders and made sure they all follow the standards I made. Waiting on a single StackOverflow answer to fix one thing and then all my spiders are ready to go. Which is really exciting I am finally finished with that.
  7. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    7/3/2020 and 7/4/2020 -Technically all of my spiders work now I am just doing some TLC on anything I went skimpy on to ensure each one works the same way as the next. -I created guidelines that I want each spider I created and future spiders to follow so each is outputting the same data in the...
  8. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    That's actually really smart thanks for the suggestion man.
  9. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    On an unrelated note found out my company hired someone for the position I was to be promoted into in a couple of months. There is already someone in that position that is objectively completely incompetent but has a really impressive academic resume and seniority so they manage to hold on...
  10. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    Perhaps. I wouldn't say the profits get smaller with more people but it is possible somewhat. If my ultimate goal is a large sellout wouldn't the lower subscriber business model limit my ultimate goal?
  11. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    6/30/2020 and 7/1/2020 -Was able to finish every other spider but one. Some of the websites share the same API making scraping super easy. -Once I finish this last spider I am going to go through and confirm the functionality of each spider before moving on to cleaning all the data which should...
  12. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    Unfortunately, I was working 16s for my job the last week in Minnesota. I will be back on the grindstone today!
  13. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    6/20/2020 and 6/21/2020 -Added another spider to scrape a website to add data to the database. -At this point, I have 10/16 spiders made to scrape the websites I need for my minimum viable product. Steps once I complete writing the spiders: 1. Finish the data cleaning pipeline which shouldn't...
  14. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    I am very positive people are willing to pay for it because they already do and at a rate a lot more than $50/month. Just not in my country. Up until recently, this method was not possible due to regulations in the U.S. I have a lot of people I know and acquaintances interested in the product...
  15. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    6/18/2020 -wrote another spider for a different website to add to my data I think I need about 4 more to have enough sources for my minimum viable product -I realized I was approaching it wrong doing one spider at a time, now I have every spider template made so when I have a problem on one I...
  16. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    6/17/2020 -Received final logos -Worked on building another spider to scrape a related website
  17. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    6/16/2020 -Didn't have much time worked from 6AM to 10PM looked over some code.
  18. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    6/15/2020 -Worked 7AM - 830PM for my job so I didn't have much time. -I took the answer from stack overflow and fixed my code so now it processes items through the scrapy pipeline. -Went through the final logo revisions with freelancer
  19. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    6/14/2020 -traveling to North Dakota for work -Asked a question on stack overflow for my code
  20. Busch_Jager

    Process thread: Subscription website

    HOT! EXECUTION 
    6/13/2020 -Was able to get my code scraping great with no issues.
Back
Top Bottom