Massive Scale Cross-Border Amazon Arbitrage

[GoldFibre]

Hi guys,

A few weeks ago I started working on my latest project, and I will chronicle my progress here to help keep me more accountable.

The project is to use a lot of automation and a bit of logistics to do retail arbitrage between Amazon regions on different continents. This is not a new business idea, and actually there are about 20-30 players doing the same thing in my market. Some of them I even know personally and professionally. But I think I can create an advantage because none of them are technical and they have to outsource their development, while I think I can do it much better myself.

Basically I have to write automation software that manages millions of listings on Amazon to maximize sales. Additionally, my logistics costs have to be low enough that I can compete.

Three weeks ago, starting from nothing, and having never used Amazon's Selling Partner API before, I sketched some architecture and started work:

• In the first week I made something that searched and profiled new products to add. I also sourced logistics partners and set up SOPs
• In the second week I did the automated listing creation and launched it last Thursday night
• And this week I optimized the throughput of the system, fixed lots of bugs, and created some UI for myself to monitor things

There are five workers running in parallel that perform one step of the process and the push to the queue for the next one. The real challenge here was the Selling Partner API because it is not well-documented and it is not standard in any way. I found many surprising issues while working with it. I was happy each time I found one of these nice Easter eggs, because it meant another barrier to entry for potential competitors. My perspective is that the worse the API is, the better the result will be for me when I figure it out.

Anyway, I'm currently measuring my progress with the number of 'BuyBox Winners', which means that my listing is the featured listing on the page, and it is actually visible to potential customers.

On Monday I had 142 BuyBox Winners. Right now (Saturday night) I have 27,567 BuyBox Winners, with 20k of those added today.

And I have one sale so far, which is nice, but that isn't the focus right now.

Maybe 27k sounds like a lot, but this is on 126k listings and I understand that my competitors have 2-3 million. So I'm not at their scale yet.

My goal for the week (by tomorrow) is to hit 50k BuyBox Winners. The system is on pace for that in its current state. So tomorrow I will do some cleanup of my codebase and make a plan for next week.

 

Dislike ads? Remove them and support the forum: Subscribe to Fastlane Insiders.

[BizyDad]

Welcome to the forum.

Looking forward to your posts.

 

[theag]

Sounds very interesting!

Since you mention logistics, I assume you will actually stock some products yourself instead of just dropshipping?

Did some quick research and found tools like PriceYak - the Original Dropshipping Automation Software. I assume that the competitors you mention probably use something prebuilt like that?

Seems like a business model driven by efficiency and whoever can afford the smallest margin will win. So having your own software definitely helps. I'd guess that the auto-ordering will be the bottleneck at scale.

 

[GoldFibre]

Since you mention logistics, I assume you will actually stock some products yourself instead of just dropshipping?

Roughly, phase two of the the business plan is to hold inventory on products that will move fast. So dropshipping gives me the data to focus my efforts on what to buy wholesale, which brands to negotiate exclusive distribution rights with, or maybe even what to white-label.

Did some quick research and found tools like PriceYak - the Original Dropshipping Automation Software. I assume that the competitors you mention probably use something prebuilt like that?

Yes, there are a number of automation softwares out there, but the ones my competitors are using are specialized to this particular scenario - cross-border (more complex logistics and costs) and massive scale (millions of SKUs).

Amazon also changed the rules around dropshipping recently and this has raised the barrier to entry. Now you have to relabel/repackage your shipments to make it clear that it is coming from your company. So the model that PriceYak has on their website (shipping direct from another store to the customer) will get you banned by Amazon.

Seems like a business model driven by efficiency and whoever can afford the smallest margin will win. So having your own software definitely helps. I'd guess that the auto-ordering will be the bottleneck at scale.

Building the software myself eliminates a lot of fixed cost initially. And I will do the manual procurement process myself until it becomes too burdensome, and then will automate it. Worst-case I can outsource it.

 

Dislike ads? Remove them and support the forum: Subscribe to Fastlane Insiders.

[Andy Black]

Welcome to the forum @GoldFibre. I see you're already an INSIDERS and posted an interesting insight in the Earnout thread. I'm curious about your background and end goal as you've clearly got IT skills and also work for a company that buys companies.

As for this project, I love massive scale stuff being ex IT myself. I like niches where I can load thousands or millions of keywords and ads and start buying data and then leads.

I worked for a company doing PPC arbitrage 10+ years ago, and having millions of keywords and low bids was how we made it work due to the low margins.

I was happy each time I found one of these nice Easter eggs, because it meant another barrier to entry for potential competitors. My perspective is that the worse the API is, the better the result will be for me when I figure it out.

Great attitude. Problems become barriers to entry. I rub my hands with glee when things don't work and I have to figure it out. That's one more hurdle between me and the competition.

I'm currently measuring my progress with the number of 'BuyBox Winners'

It's great that you're measuring progress.

I have one sale so far, which is nice, but that isn't the focus right now.

This is cool. I do lead-gen now and when we generate a lead we often just see that as the systems working end-to-end, while we keep building out campaigns etc.

 

[GoldFibre]

Welcome to the forum @GoldFibre. I see you're already an INSIDERS and posted an interesting insight in the Earnout thread. I'm curious about your background and end goal as you've clearly got IT skills and also work for a company that buys companies.

As for this project, I love massive scale stuff being ex IT myself. I like niches where I can load thousands or millions of keywords and ads and start buying data and then leads.

I worked for a company doing PPC arbitrage 10+ years ago, and having millions of keywords and low bids was how we made it work due to the low margins.


Great attitude. Problems become barriers to entry. I rub my hands with glee when things don't work and I have to figure it out. That's one more hurdle between me and the competition.


It's great that you're measuring progress.


This is cool. I do lead-gen noe and when we generate a lead we often just see that as the systems working end-to-end, while we keep building out campaigns etc.

Hi Andy,

My background is logistics & software development and my end goal is to not have to work full time, so that I have the freedom to choose how to spend my time.

This business might have a PPC aspect later using Amazon ads and it would be an interesting automation challenge. Considering these listings are on relatively slow-moving items, it would probably need low bids, like what you did with the PPC arbitrage.

 

[GoldFibre]

My goal for the week was to hit 50k BuyBox Winners. The automation should cruise to that while I'm asleep tonight.

Today I did some competitor analysis, checking out their pages and catalogs. Here are a few takeaways:

1. I almost never beat the Amazon global store, even if my price is lower
2. The number two competitor after Amazon US is a company I know and is definitely the most organized of the bunch. They have in-housed operations and even development but their business is more than just Amazon.
3. There are about eight other competitors with over 1 million listings
4. Besides competitor number two, the reviews are pretty abysmal and it looks like very little effort has gone into customer service

Here's the chart of how I fare against the most common competitors:


It looks like I'm winning a lot, but I took a second cut, and filtered by products that have a high sales rank, meaning they sell more often.

This paints a different picture:


My win rate is a lot lower here, where it counts.

So I've decided my next KPI to track is:

• Fast-Mover Winners: BuyBox Winners on products with display group sales rank less than 5,000, excluding anything where I compete with Amazon US/UK

Here are the current stats:


I'll make my goal to get this number to 1,000 by the end of the week through a combination of cost optimization and adjusting my automation to discover and list more fast-movers.

 

Dislike ads? Remove them and support the forum: Subscribe to Fastlane Insiders.

[GoldFibre]

+1 new sale last night

Progress:

• Added UI for tracking my goal progress and inspecting the fast moving listings and winners
• Added UI for adding new search keywords, then used it to double the number of keywords that get automatically scanned
• Reached out to five different courier companies to try to get better pricing on the domestic leg and got about 12% lower cost
• Created a collaborative Google Sheet with my international logistics vendor to coordinate on order tracking
• Discussed with the vendor on SOPs and considerations for the next market

We went from 60 Fast-Mover Winners yesterday to 239 today, which puts me on pace for this week's goal. I'm not sure the exact reason it went up so much, especially since most of this happened before I made the changes listed in the progress section.

The order tracking Google Sheet is a temporary solution, and I'll build some custom UI for this once the order volume picks up. At this point it is a better use of my time just to do a spreadsheet in 10 minutes and then spend the rest of my time optimizing the listing automation.

 

[GoldFibre]

Surprise! I have zero Fast-Mover Winners. I actually have zero BuyBox Winners at all.

Right after I got three(!) sales in the morning Amazon automatically set my account to 'holiday mode' to block new orders. This is because I had too many unfulfilled orders as a new seller and they want to make sure I deliver them within their SLA before reopening my account. Since it will take at least a week to deliver from overseas, I won't have any more sales or BuyBox Winners this week.

And now that this week's goal is effectively impossible to achieve, I'm switching it from 1,000 Fast-Mover Winners to 20,000 Fast-Mover Listings. So when my account eventually comes back from holiday it will have many more good listings ready and waiting.



Progress:

• Discussed w/ Amazon account manager about 'holiday mode' and also about profiling strategies to reduce the risk of IP infringement claims - I already have one on my account
• Added a 'brand blacklist' feature to my bot to help with the IP infringement
• Got more domestic delivery quotes and lowered the cost a further 15%
• Set up a call with the delivery provider to sign the final contract and get their web portal access - I need this to generate the package labels
• Refined my delivery cost estimation code to handle the lower cost as well as the weight breaks
• Improved the order fulfillment Google Sheet with some additional columns requested by the international logistics partner
• Signed up for an Amazon Prime business credit card, which gives 5% cash back on the first 150k USD of spend at Amazon.com. This is the best credit card to lower my procurement costs
• Doubled the number of keywords in rotation to help reach my goal

 

[GoldFibre]

My automation is running off of a laptop in my home, and this morning I forgot to check it before heading to my full time job. When I got there and checked the web interface I realized that the bot had stopped running, and I don't have a way to restart it remotely. So I lost some time there and learned a lesson.

Progress:

• Discussed operational SOP with delivery partner
• Got my Amazon 5% cash back credit card approved and applied to all new purchases
• Handled a couple of bugs and exceptions that I found
• Increased search keywords to 800 using Shopify's product taxonomy

 

Dislike ads? Remove them and support the forum: Subscribe to Fastlane Insiders.

[GoldFibre]

In order to reduce the risk of getting my account suspended for IP violations, I built a worker that scrapes a global trademark registration site to determine which brands I should avoid listing against in my jurisdiction. This was on the advice that my account manager gave me earlier in the week.

Then later this morning I spoke to someone more senior in Amazon who told me that wouldn't be necessary, and they can get my account unsuspended quickly if I ever ran into trouble. I'll continue to run this worker to at least capture the data, because it might come in handy later.

Progress:

• Built trademark registration worker
• Fixed a number of little bugs
• Added more search keywords up to 1,000

 

[BizyDad]

Gotta be honest. This thread is giving me big get rich quick vibes.

I love how honest you're being about the problems you're facing. I love even more how quickly and creatively you are solving them.

"Let me scrape a database of trademark and get region specific" said no one ever. Lol.

I'm curious a little bit more about you. Can you tell us about your background? How did you get so good at this?

 

[Saad Khan]

Hi @GoldFibre, awesome progress there! You're definitely onto something big if this works out.

I was wondering whether you use Keepa API to determine sales rank for listings you're analyzing? Plus I see that you haven't talked much about what criteria you go through to determine buybox winners? Perhaps I give you feedback on how to further refine your criteria? I haven't worked in amazon arbitrage but have a client working in wholesale FBA. Our biggest challenge yet has been to source the products from profitable and authentic sources with all the documentation and invoices. We have been traditional in our approach so far.

Pro tip:

If you're looking to gather a list of top 100 US Amazon sellers doing wholesale/arbitrage, consider investing in smartscout. They have a big list for every marketplace that you'd be interested in scraping.

 

Dislike ads? Remove them and support the forum: Subscribe to Fastlane Insiders.

[Saad Khan]

Gotta be honest. This thread is giving me big get rich quick vibes

So is it a good thing or a bad thing?

I think he is onto something and he did acknowledge that there's 20-30 people doing it.

 

[GoldFibre]

Gotta be honest. This thread is giving me big get rich quick vibes.

I love how honest you're being about the problems you're facing. I love even more how quickly and creatively you are solving them.

"Let me scrape a database of trademark and get region specific" said no one ever. Lol.

I'm curious a little bit more about you. Can you tell us about your background? How did you get so good at this?

Thanks for the feedback - I think this business is a good match for me, because I have a logistics tech background and I also have a strong personal network in the industry. So I can get valuable information directly from Amazon and can get preferential rates and setups from vendors that an unknown person might not.

In the past I've worked both as a developer and as a product manager, so maybe what makes me look 'good' at this is that I am always trying to focus on the most impactful thing to work on, in the most efficient way.

I've failed at a number of other businesses I've tried in the past, such as automated day trading and POD eCommerce. But this Amazon idea has a few more things going for it, so I think I have a better chance of success.

Hi @GoldFibre, awesome progress there! You're definitely onto something big if this works out.

I was wondering whether you use Keepa API to determine sales rank for listings you're analyzing? Plus I see that you haven't talked much about what criteria you go through to determine buybox winners? Perhaps I give you feedback on how to further refine your criteria? I haven't worked in amazon arbitrage but have a client working in wholesale FBA. Our biggest challenge yet has been to source the products from profitable and authentic sources with all the documentation and invoices. We have been traditional in our approach so far.

Pro tip:

If you're looking to gather a list of top 100 US Amazon sellers doing wholesale/arbitrage, consider investing in smartscout. They have a big list for every marketplace that you'd be interested in scraping.

Thank you for the tips and insights.

I'm not using any third party APIs or tools right now. They will most likely use the official SP API under the hood, which is what I'm using directly. After some initial struggle, now I'm working very efficiently with the API and can add features quickly.

The BuyBox winner is determined by Amazon and they return this information in their API. The criteria is not published, but it looks to be some combination of price, transit time and seller rating. Sales rank is also directly returned by the SP API. There are two ranks: display group sales rank and category sales rank. Amazon does not publish the actual number of sales though.

I'm only sourcing from one Amazon region to another, so I'm assuming that product authenticity shouldn't be a big issue.

 

[GoldFibre]

Some of my orders are arriving tonight, so I can start fulfilling and getting my account unblocked. Probably it will take until mid next week for the last order to be delivered.

I discovered an interesting problem - I cannot trust that the ASIN (Amazon's version of a UPC) represents the same product across Amazon marketplaces. It matches most of the time, but sometimes it is different. For instance, I found one ASIN that is a dog toy in both marketplaces, but in one it is a stuffed bull and in another it is a stuffed duck.

This weekend I'm going to look for solutions to this, because if someone orders a duck and receives a bull, they are likely to complain and my seller rating will go down. I saw a lot of complaints like this in my competitors' seller accounts, so I think that solving this can be a competitive advantage for me.

Progress:

• Reduced db load that was causing table locks and slowing down the bot
• Fixed logic in the updater that was skipping updates for some listings
• Spoke with my international logistics vendor to make absolutely sure that my orders are going to arrive in enough time to meet Amazon's SLAs

 

Dislike ads? Remove them and support the forum: Subscribe to Fastlane Insiders.

[amp0193]

Been following this thread out of pure curiosity.

I'm fascinated.

Keep the updates coming.

 

[Saad Khan]

I discovered an interesting problem - I cannot trust that the ASIN (Amazon's version of a UPC) represents the same product across Amazon marketplaces.

It also goes the other way around (one UPC for multiple products) so I'd recommend you configure the bot to validate both the ASIN and product identifiers. In most cases it's a UPC code. Sometimes its an EAN code too

 

[GoldFibre]

It also goes the other way around (one UPC for multiple products) so I'd recommend you configure the bot to validate both the ASIN and product identifiers. In most cases it's a UPC code. Sometimes its an EAN code too

Thanks, I'm working on this now and will add a check for other identifiers as you suggest.

 

Dislike ads? Remove them and support the forum: Subscribe to Fastlane Insiders.

[GoldFibre]

It's the weekend and I had a lot more time to work on the business. I reviewed many, many different listings made by the bot and found opportunities to improve the pricing and give more accurate updates.

There are also some processes for price correction that you absolutely have to do through Amazon's seller portal, because there isn't any API for it. Luckily these only affected about 100 out of 200k listings, so the manual work isn't a big burden. If it becomes a burden though, then this could be done later through browser automation.

I added a check that the names of the ASINS match across marketplaces and sent any mismatch to manual review.

At @Saad Khan 's suggestion I also added a check for EAN and UPC mismatches.

Then I made a UI for me to review anything that is caught by these checks and looked through what it was picking up.

After reviewing I found lots of false positives with name mismatches. In addition, my example of the duck/bull dog toy won't even be caught by this because all the identifiers match, and even the model number matches. The real problems with the listing are the listing title (different from product name), the description and the images.

So after all this investigation, I removed the name as a criteria for a hard stop, and I added in the model as a third identifier, alongside EAN and UPC.

But after running the bot for several hours, not a single mismatch was found. So I haven't really solved any problem with all this work.

On the fulfillment side I scheduled a pickup for the first package, but ran into issues. When the order was initially placed my account was registered with Easy Ship, but in the meantime I asked Amazon to switch my account to Self Ship, and the Easy Ship settings disappeared from my account. So in the end, the pickup was arranged to the wrong address and Amazon wasn't able to change it.

We will have to make a second attempt on Monday, which is the latest day for this order as per the Amazon SLA.

Three more orders are arriving this evening, and those will be picked up by my own logistics partner on Monday, since the warehouse will be closed Sunday.

Then I have my straggler - the final order to fulfill before my account gets unblocked - and it will only arrive in my US facility on Monday. So I want to push my vendor to dispatch it on the same day if possible. Each day I lose here is another day my account stays blocked.

Progress:

• Automated the import of the keywords from the Shopify product taxonomy. Went from 1,100 keywords to 5,700 after spending 10 minutes on a script. Before I was spending an hour each day inputting a few hundred of these manually!
• Made some mismatch detector that isn't helping yet
• Refactored my list pages into a common component - This is in preparation for making a number of pages in the UI that highlight potential problems for action, such as an unusual weight or a large price difference with my competitors
• Disabled some cron jobs that were too expensive on the DB
• Started thinking about how I want to manage orders

 

[GoldFibre]

So I didn't come near my goal for the week. Ultimately I found that there were more important things to work on and I intentionally didn't focus on the goal after the first two days. Still, we were only at 516 Fast-Mover Listings at the start of the week, so there was more than 15x progress. Hopefully this means that when my account comes back online I will get at least 15x sales!

Today I started the day by optimizing my offer updater automation because I found that some offers haven't been updated in a long time. I think that the oldest updated offer is probably a good KPI to track, because the more frequently I update the less likely I am to get an order for something that has gone out of stock in the source marketplace.

My primary DB table has grown so large that I can't rely on any complex queries or even simple counts on it anymore. So I converted everything into queues stored in separate tables. After this change I no longer have any DB locks and every query is back down to a few milliseconds.

Even after that my listing updater was running too slowly, and so I found one more opportunity to parallelize. At the current rate I can update about 11k listings per hour, but with now around 400k listings, it means it would take 36 hours to update every one, if I updated them in a single queue. I do have priority queues, but still it is taking too long.

I think for next week I will have two goals - one for each half of the business:

• Automation: Oldest Lowest Price Offer < 24h - not using 'BuyBox Winners', because my account is still blocked so I have no winners
• Fulfillment: Deliver all 5 pending orders to get account unblocked - this largely depends on me pushing my partner to expedite the last package, which only arrives in the US facility tomorrow

I put both of these goals on the home page of my UI, so that I'll see them whenever I start working.

For the automation goal, I need to start by adding some telemetry to this worker, so I can see where the bottleneck is. After that it is mostly async techniques of batching and parallelization. I've done a lot of this already and I think there are still more opportunities. There is a theoretical maximum speed based on the rate limits of the SP API, so I will try to estimate that and then get closer to it.

One way to scale horizontally is to have additional seller accounts, because each account has its own independent rate limit. Then I could split the products between however many accounts I have, multiplying the update frequency. To get more seller accounts I would need to register more legal entities, which takes a few weeks and requires some cash outlay. If things go really well and the increased benefit appears to be worth the cost, then I may pursue this later.

For the fulfillment goal, I will tell the partner I want it leaving on a Monday consol if they have one. And if they don't then I'll ask them to ship it direct to the end customer by international courier and I'll pay the cost.

 

[theag]

What are you writing this whole thing in? Would love to hear a bit about your tech stack and setup. That primary db table definitely blew up fast!

Getting around API limits through multiple legal entities sounds like a PITA, but creates a nice barrier of entry too.

 

Dislike ads? Remove them and support the forum: Subscribe to Fastlane Insiders.

[GoldFibre]

What are you writing this whole thing in? Would love to hear a bit about your tech stack and setup. That primary db table definitely blew up fast!

Getting around API limits through multiple legal entities sounds like a PITA, but creates a nice barrier of entry too.

I'm writing it in TypeScript on Node, because this is what I'm most productive with.

The DB is MySQL, because it's free. It's hosted on a spare laptop in my home right now, also because it is free.

This could be done in any language and with any DB. The challenge here is really with the SP API structure and rate limits, and trying to squeeze the most out of them.

 

[GoldFibre]

After six calls with Amazon seller support, about seven emails with various Amazon departments and finally a call with a delivery driver, I was able to get my first package picked up.

And the second it was picked up, my account got unlocked!

I thought I needed to deliver all five of my orders, because that is what the automated email from Amazon had said, but it turned out to not be the case. I haven't even delivered a single one - I only had the first one picked up.

So I'm going to drop this goal, since the above one was only there to get my account unlocked.

Today I went to my destination warehouse and had some discussions around SOPs with the ops team, and then met with the GM and spoke about a few suggestions for improving the transit time. I also learned a few more filters I need to put on my listings at some point.

I added some basic telemetry to my offer workflow, which narrowed down the performance bottleneck a little bit. But I didn't have a lot of time for coding today. Tomorrow there should be more on this side of things.

 

[Two Dog]

I've only heard about retail arbitrage with local stores providing the inventory. It sounds like you're trying to exploit regional pricing differences and somehow pocketing money. If purchases are being made from Amazon sellers (???) and have to be shipped directly by your store, it doesn't seem like there would be enough margin to arbitrage.

What's the 10 year old child summary of what you're doing?

 

Dislike ads? Remove them and support the forum: Subscribe to Fastlane Insiders.

[GoldFibre]

I've only heard about retail arbitrage with local stores providing the inventory. It sounds like you're trying to exploit regional pricing differences and somehow pocketing money. If purchases are being made from Amazon sellers (???) and have to be shipped directly by your store, it doesn't seem like there would be enough margin to arbitrage.

What's the 10 year old child summary of what you're doing?

This is my best try at a 10 year old child summary:

I have a robot that finds products in the US and puts them on Amazon sites in other countries.

The products aren't available in those countries, and some people are willing to pay more to get them.

I save money on shipping by combining small packages together into one big package.

 

[Two Dog]

This is my best try at a 10 year old child summary:

I have a robot that finds products in the US and puts them on Amazon sites in other countries.

The products aren't available in those countries, and some people are willing to pay more to get them.

I save money on shipping by combining small packages together into one big package.

That's perfect! Thx for taking a moment to write it out.

I never would have thought other countries would pay more than US buyers or there would be enough margin with international shipping to make it worthwhile. That was a big part of my blind spot in understanding your business model.

 

[GoldFibre]

I haven't made any optimizations to my bot - I've only let it run 24/7 and we are getting much closer to the goal.


My account got reinstated yesterday and I got two orders in the morning. And then my account got automatically blocked again for having too many outstanding orders.

This might be a good thing, because each of those two orders revealed some deficiencies in my automation:

1. I got an order for two of the same item, but when I went to procure it only one was in stock in the US
2. I got an order for an item that had a shipping time in Amazon US of 15 days, which would make it impossible to hit my Amazon SLA

So I need to account for both of these factors in my own offers and coding it shouldn't be too complicated.

I was supposed to have three orders picked up for final delivery today, but my delivery partner ghosted me. They didn't answer my phone calls and then in the late afternoon they sent me an email that they 'need more time'.

Luckily I have an offer from several other vendors, so I called one of them and told them they have a chance to win the business. I tried to push them to pick it up today, but it was too late in the afternoon. So hopefully they will pick it up tomorrow.

I expect that once these orders get picked up and I confirm them in Amazon's system, they will reopen my account again.

I was super busy with my full time job and other stuff today, so didn't have any real time for coding.

 

Dislike ads? Remove them and support the forum: Subscribe to Fastlane Insiders.

[GoldFibre]

Today I got my account set up with my backup delivery partner and scheduled the pickups for three of the orders. As soon as I confirmed the orders with Amazon they unrestricted my account again.

The backup delivery partner is also about 5% cheaper than my previous one, so I'm also saving some cost.

I reached out to the customer whose order I cannot fulfill on time and informed them of the situation. I told them that they can cancel or they can wait an additional three weeks in the worst case.

I'm hoping that they cancel the order, because then my seller rating won't be hit by a late delivery.

I started to work on incorporating the Amazon US transit time into my own, and it looks like this information isn't returned by the SP API, unless you are querying your own listing. So maybe I need to resort to web scraping, which with Amazon is notoriously hard.

It's the same story with trying to get the available quantity. With the SP API you only know if something is in stock or not, but you don't know how many are available.

Anyway, I started messing around with web scraping and it is going to be a lot of work. Here are the issues I know of so far:

• Amazon is very aggressive in banning web scraping bots, and I'm not sure if the precautions I'm taking will be enough
• There are obfuscation techniques in the backend calls, HTML and JS to make it more difficult to scrape
• The site doesn't work properly without JS enabled
• There are many different product page templates that will need their own script variation to scrape
• The transit time could be different, even longer sometimes, if you are using Prime to purchase, which I am. But I can't log in as myself and then operate a bot or my business account would get blocked
• Some product options, such as clothing sizes, do not get selected automatically, even if the ASIN in the URL is tied to a specific option. So I have to figure out a way to reliably select the correct option before checking the transit time and quantity

There are third party tools available for this, but the ones I found aren't designed or priced for continuously scraping 100k's of products every day.

I'll hammer at it and try my best to get something at least partially working by this weekend.

I also have a sort of nuclear option of excluding any product not fulfilled by Amazon in the US, so I won't get anything with exorbitant transit times. I could do this first and then add back non-FBA products once I find a good way to capture the transit time.

 

[GoldFibre]

I thought some more about the difficulties in getting exact transit times via web scraping. Whenever there is a complex problem in programming, there is usually a 'stupid' solution that generally solves the problem extremely quickly while sacrificing some other aspect, like efficiency or cost.

Here is my 'stupid' algorithm that doesn't require any web scraping:

• If the US seller is Amazon or FBA, add X days transit time
• If the seller is not Amazon or FBA, add Y days transit time
• Y will be much larger than X

This will overshoot the actual transit time at the cost of lowering my revenue, but it will reduce the risk that my account gets blocked for failing to deliver on time. And getting blocked is my number one risk in this business so I take this tradeoff for now and monitor things.

The 'stupid' solution saves me maybe months of effort trying to get the exact answer via web scraping, and also avoids increasing my costs and risks (renting proxy servers, risk of banning).

So I implemented this in about ten minutes, after spending several hours on a web scraping solution last night.

Then I got back to optimizing my offer updater - my goal for the week.

• Switched two parallelized calls to leverage a cache
• Skipped patch updates in more cases

After these changes I went from about 230 offer updates per minute to 295 per minute. Not sure if it will be enough to get to my goal though. I have one more idea to speed things up that I will try tomorrow.