![aws lambda webscraper aws lambda webscraper](https://d2908q01vomqb2.cloudfront.net/fc074d501302eb2b93e2554793fcaf50b3bf7291/2021/03/29/Pupeteer-image.jpg)
Of course, any lazy serverless developer will use Stackery to avoid the tedium of writing their own YAML and configuring permissions, so naturally, I built this scraper in our app: Make sure they exist and your bucket is in the same region as this function.` remove emojis as they don't show up in the output HTML :(Ĭonst regex = /(?:|(?:\ud83c). options for cheerio and request-promise Here's the scraper function I built that does the last part of the process: const AWS = require('aws-sdk')
Aws lambda webscraper manual#
That doc is then reviewed a few times a week and the most interesting items are written up in our public Changelog (this is the unavoidably manual part of the process, until we build an AI that writes puns, anyway).When the PR is merged, a scraper Lambda function is triggered by a Git webhook and compiles the new Changelog item into a single doc.Each engineer updates an internal Changelog file in an individual repo as part of the PR process.Here's what our new Changelog process looks like, in a nutshell: A better way of doing thingsĪfter much thought and some trial and error, we came up with a better system.
Aws lambda webscraper update#
However, the initial Changelog update process was anything but lazy:
![aws lambda webscraper aws lambda webscraper](https://cdn.nerdyelectronics.com/wp-content/uploads/2020/01/WhatisAWS-300x187.png)
One of our more recent processes was adding a Changelog to our Docs page, to let our users know what's going on with Stackery and so we can announce major updates. I mean working smarter rather than harder, and finding ways to automate repeated processes. That doesn't mean programming from the couch in your PJs (though that's what work from home days are for, amirite?). I am a firm believer in lazy programming.
![aws lambda webscraper aws lambda webscraper](https://adtmag.com/articles/2017/03/23/~/media/ECG/adtmag/Images/2017/03/aws_lambda_s.png)
If only I could also make robots mow my lawn. Luckily, as an engineer, my laziness is an asset - because this week, it led me to write an HTML scraper for our Changelog, just so I wouldn't have to manually update the Changelog feed on our new app homepage (btw, have you seen our new app homepage? It's pretty sweet). Especially if it means I can avoid doing the same thing more than once. This means that if I can automate a task, I will absolutely do it.