Posts
Tech & Innovation Blog
I’m a start-up founder and angel investor, and here’s why you should consider a tech career at ADP
Voice of Our People, Why ADP, Workmarket
We recently caught up with Jeff Wald, angel investor, and President and Co-Founder of ADP’s WorkMarket.
We asked him why someone should consider a career in technology with ADP. Here’s what he said:
“What may have started life as a payroll company has transformed into one of the leading tech companies of the 21st century.”
When ADP acquired WorkMarket in 2018, a lot of our engineering and product associates were curious about how a ‘payroll company’ such as ADP aligned with WorkMarket, a cutting-edge start-up. It’s a fair (and expected) question, and one we continue to answer with ease as we seek the best and brightest in tech talent to join the ever-growing ADP team.
As part of the onboarding into ADP, WorkMarket associates quickly discovered the scale of ADP’s tech workforce and the market-leading technology the organization builds continuously. The WorkMarket team realized that ADP was the bridge between the cutting-edge nature of a tech start-up and the opportunity to have that technology utilized by hundreds of thousands of companies around the world. It’s a scale that few companies can even imagine.
What may have started life as a payroll company has transformed into one of the leading tech companies of the 21st century. With a significant emphasis on how to provide a better experience for users utilizing the latest technology, such as machine learning, AI, and conversational UIs, ADP offers its product and engineering associates the excitement and leading-edge environment of a start-up.
At ADP, tech associates get to work for a company that knows security and scale better than anybody. You get to work with a massive team of 9,000 technologists around the world. You get to work with one of the largest business mobile apps in the world. You get to be a part of a team that is moving trillions (yes, trillions) of dollars around the globe. In fact, ADP has built the largest business app ecosystem, ADP Marketplace, where ADP is the connection point for hundreds of companies to interface with hundreds of thousands of clients.
Our collaborative environment also means your opportunity is limitless. As a technology associate joining ADP, you can widen your purview beyond your specialization, and have the chance to exchange knowledge with the thousands of other technologists at ADP.
And it’s not just about the technology. It’s not an understatement to say that since WorkMarket joined ADP in 2018, I’ve been blown away by how truly partnership-oriented the people are here. The culture and environment at ADP are incredible! Highly mentorship-oriented, you can learn as much as you are willing to take in. There are always people ready to help you at ADP and opportunities to expand your skills and relationship base. This, combined with ADP’s cutting-edge technology and sheer size, means that if you want to work with a company that has the breadth and scope of opportunities to influence the world, ADP is that place.
There are only a handful of technology giants that are truly changing the world, and there are only so many companies on earth where the things you build touch tens of millions of people. Where you create something that substantially impacts the way we live. You know the names of those other tech giants, but you may not have known ADP was one of them. We are just getting started! Join us on this journey.
Jeff Wald is a President and Co-Founder of ADP’s WorkMarket and the author of The End of Jobs: The Rise of On-Demand Workers and Agile Corporations. He is based in New York City.
Tech & Innovation Blog
Time clocks in the age of COVID-19: How ADP developed a (mostly) touchless solution in just a few weeks
Pandemic, Innovation, Voice of Our People
In early March 2020, as the COVID-19 outbreak started to expand throughout the U.S. and Canada, the team working on ADP’s new Time Kiosk system started getting the same question from many of our clients: “Is there a way to use this without touching it?”
By Jonathon Gumbiner, Senior Product Manager
In early March 2020, as the COVID-19 outbreak started to expand throughout the U.S. and Canada, the team working on ADP’s new Time Kiosk system started getting the same question from many of our clients: “Is there a way to use this without touching it?” At the time, we were several months into a pilot program for the tablet-based timecard management app with more than 1,000 clients.” But most of them hadn’t adopted the app’s facial recognition feature, instead opting to tap in their badge number. And even for those who did use facial recognition, Time Kiosk still required each worker to tap the screen—a suddenly dangerous proposition during a global pandemic.
For companies that had essential workers on-site, we suggested an immediate but imperfect solution: low-cost touch pencils each employee could use to navigate the app. But we knew we’d ultimately need an integrated, turnkey option—and we’d need it as soon as possible.
After a quick brainstorm, we narrowed in on the fix that seemed most promising: First, we’d reconfigure the app to perform facial recognition by default, whenever someone was in front of the camera. Then, we’d use the tablet’s built-in virtual assistant, which powered features like Siri and Google Assistant, to respond to voice commands within Time Kiosk. If we were successful, employees would be able to start a workday, take lunch and other breaks, and clock out, all without touching the screen.
Within a couple of days, our developers were able to build a rough proof-of-concept. It was clunky and far from intuitive—to clock in, for example, you had to say “tap clock in” instead of simply “clock in.” But it was enough to help our senior leaders understand our vision for a more-refined solution—one that would meet the high standards we’d set for the original Time Kiosk experience. We got their buy-in and started to build.
Voice recognition was the first challenge. For one thing, as anyone who’s used a virtual assistant knows all too well, there are phrases it just won’t recognize. Also, in order to release the touchless features as part of Time Kiosk’s formal launch in both the U.S. and Canada, which was just a few weeks away, we needed to develop voice recognition for not only English but Spanish and French, as well—languages no one on the team speaks. Thankfully, as a global company, our partners from other ADP teams came to our rescue, helping us quickly create a repository of words to which the tablets would reliably respond.
Of course, we couldn’t make every action completely touchless. Switching between an employer’s custom job or department codes, for example, would require an employee to scroll through options that voice recognition likely wouldn’t cover. But what we could do was keep people informed. With the help of our UI team, we developed a treatment to add an icon for every touchless function, so employees could see at a glance whether they’d need to touch the screen. If so, they could wash their hands or take other precautions before they acted.
Once we’d finished the first phase, though, we came to a larger challenge: quality assurance. We spent twice as much time testing the new touchless features as we’d spent building them, going through every single action a user could take to make sure we’d identified everything properly. Because voice recognition touched the entire product, we had to review it all—and quickly, requiring a true team effort from QA. What’s more, we happened to be in the middle of transitioning to a new UI, so we needed to test both the current and the incoming interfaces, making the process twice as long.
Yet perhaps the biggest challenge of all was the pandemic’s impact on how it got done. To make sure the new features worked well in all three languages, we needed service reps and tech partners to help us with testing. But most of ADP’s 58,000-person team was working from home. I couldn’t simply walk upstairs, hand someone my tablet loaded with the latest version of our work and ask them to play around with it for a while and bring it back at the end of the day. Instead, we had to find a way to get it on their tablets remotely—no easy task given the tablet’s security restrictions. Luckily, our team was able to build a package and set of instructions that I could share, allowing partners to offer live feedback via an embedded diagnostic tool. They were invaluable in helping us fine-tune, especially our translations.
In the end, thanks to the hard work of everyone on the Time Kiosk team and many of our colleagues, we were able to meet our goal, transforming the app into an intuitive, mostly touchless experience in a few short weeks. Like any quick-turn project, it wasn’t without a few bugs. But the team’s rapid response to client questions and weekly Q&A calls have helped us not only serve their needs and build stronger relationships with them. Time Kiosk has now officially launched, and our sales teams tell us the touchless technology has been a conversation driver with both clients and prospects.
Even after the COVID-19 outbreak has passed, we see great potential in what we’ve learned about voice and facial recognition, whether it’s better accessibility for employees with disabilities or voice biometrics for authenticating service calls. In the meantime, we’re proud to say that when our clients had an urgent need, we were able to quickly deliver a solution that works—and that’s helping keep thousands of their people safe.
Jonathon Gumbiner is a Senior Product Manager at ADP in New Jersey.
Tech & Innovation Blog
Spotlight on ADP’s Distinguished Engineer – Jigesh Saheba
Recognition, Voice of Our People, Engineering
At the bottom of Jigesh Saheba’s email signature is this quote from Reed Hastings, co-founder, and CEO of Netflix: “Companies rarely die from moving too fast, and they frequently die from moving too slowly.”
With that mindset on innovation, it’s easy to see why ADP named Jigesh Saheba as their first Distinguished Engineer.
This new honor recognizes leaders and individual contributors who are at the forefront of ADP’s transformation into a global technology leader. To be recognized as a Distinguished Engineer, one has to have achieved noteworthy professional technical accomplishments in an engineering role. Also, that individual must have created and fostered technical career paths for technologists that allow them to master their craft, innovate, and generates groundbreaking solutions that can transform the future of work.
We caught up with Jigesh shortly after being named the ADP Distinguished Engineer to learn about the start of ADP’s Marketplace and asked a few questions about his journey. Here’s what he shared:
When did you start working at ADP?
I joined ADP in 2002 with the acquisition of AtWork Technologies, a benefits administration startup in Atlanta, GA.
Can you share your career journey at ADP?
I started at ADP as the Director of Application Development for Benefits eXpert system and later promoted to Senior Director role. In 2006, I briefly joined the Enterprise Architecture team and was soon assigned the CTO role for ADP Pre-Employment Services. There I played an essential role in the acquisition, assimilation, and growth of multiple products. I rejoined Enterprise Architecture in 2010 and started working on the mobile platform. Later that year, I joined the Roseland Innovations Lab, which delivered several groundbreaking technologies, including ADP Mobile Solutions, event-driven APIs, and semantic search. I received a promotion to Chief Architect in 2013. In the same year, I started working on building an integrated HCM ecosystem around ADP, which was the birth of ADP Marketplace. In 2015, I became the VP of Product Development for ADP Marketplace.
You are basically the cornerstone of ADP Marketplace. How were you able to develop such an integral part of the solutions we offer our clients?
For ADP Mobile Solutions, ADP systems exposed their data and services via hundreds of Web APIs. I started thinking about how Web APIs can expand the reach of ADP to new channels such as third-party applications, partner networks, and integrated solutions. I believed ADP could position itself in the center of a vibrant HCM ecosystem and drive higher customer engagement. I envisioned a business model expansion and new markets. I theorized that the app stores and marketplaces, a familiar notion of consumer-mobile ecosystems, could become essential tools for enterprise application innovation, discovery, and distribution. I created a presentation to convey the concept and gained senior management approval.
We incubated the idea in the Lab and demonstrated the working platform in the summer of 2014, and then we started working with a handful of partners to prove out the model and platform. We received industry acknowledgment when ADP won two prestigious awards at the HRTech Conference in 2015. Don showcasing the award-winning ADP Marketplace is one of the proudest and most satisfying moments of my professional career.
How did you learn you were going to be named ADP’s first Distinguished Engineer, and how did you feel when you were told?
I heard about the Distinguished Engineer role and my nomination from Rich Guinness, SVP of Product Development, Shared Services. I shed tears of joy when the announcement came and was so fortunate that I could share that moment with my family in person. I am incredibly honored and grateful for the role and very thankful to GPT leadership for enabling an environment where innovation and quality engineering thrive.
What excites you most about the opportunity to be a Distinguished Engineer at ADP?
I am very honored and grateful for the recognition. I hope I set an example of the role. I am excited about the voice it offers, and I hope to amplify the voice of all technologists at ADP.
What’s a day in the life like for you and your team?
I am very fortunate to be working with some of the most talented people I know. We enjoy collaboration, and one often hears spirited discussions at “the wall,” an open collaboration space with a whiteboard wall at our Lakeview home. We are known for clapping a lot, which promotes camaraderie. The best days are when I work with the team to drive product decisions, design solutions, participate in reviews and demonstrations, and even troubleshoot issues.
What advice do you have for associates who may have ideas that they want to share?
Never give up!
What do you believe is the best reason to work at ADP?
ADP Culture – Integrity, inclusiveness, diversity, trust, and innovation.
What’s the last book you read?
Something Deeply Hidden by Sean Carroll
What is your favorite TV or streaming show?
GOT (Game of Thrones)!
What is your favorite ADP value?
Integrity is Everything
If you could only eat one food for the rest of your life, what would it be?
Ice cream. Yes, really! (Those who know me would agree)
What is something that would surprise people about you?
I started my career at Johnson Space Center in Houston, Texas, working on navigation software for the Space Shuttle.
Are you a cat person or a dog person?
Dog
Congratulations, Jigesh, on this amazing accomplishment. We’re excited to see what’s yet to come!
Tech & Innovation Blog
Nice to meet you, virtually speaking through your R.E.S.U.M.E.
Women in STEM, Voice of Our People, Career Advice
Jyotsna M. offers her upbeat and positive approach to building a resume that gets results and showcases the one and only unique you.
The sun shines brightly, and we look to a new day where the world of work is digitally and virtually connected. If you are a recent college graduate looking for an opportunity or someone looking to change the career path to try something new, this is an exciting time to research, explore, and reach recruiters.
The unfortunate side effect of the pandemic is that the world is now predominantly virtual, temporarily suspending shaking hands and making connections over coffee. It is still too early for in-person career fairs at the fancy convention centers to resume, or the opportunity to walk up to a recruiter and talk to them. The world has changed, a bit too fast for comfort. For the opportunities posted, there are a lot more applicants.
Now more than ever, your RESUME creates the first impression about you and your skills for recruiters and hiring managers. Your resume will determine if you are a “possible candidate” for an opening with their organization!
A resume is not an itemized bullet list of everything you’ve studied, where you’ve worked, or volunteered. Chances are recruiters will not have time to read, review, and recap all of that to determine your candidate eligibility. So, let your resume do the job it was meant to do!
Something to think about: When was the last time anyone read a full newspaper? Yes, every row and column on every page, to get the news? You only scan the highlights, right?
Your resume has one objective: To make highlight your skills and make an impression on a recruiter, so that you get invited for an interview! That’s it!
Your R.E.S.U.M.E. needs to highlight accomplishments that are relevant to the job posted. So, you may need to customize it if you plan to apply for jobs in different fields. Start thinking of your elevator pitch in document form. Easy, right?
It’s OK to showcase your skills. You are the only person who can present your talent. You are the builder of your aspirations, and the driver of your career.
Visualize yourself walking up to receive an award, and the greeter announcing your specific accomplishments that led to the award. This is about you, so why hide your skills in a bulleted list? Shine your skills brightly enough to be seen.
Now, start writing your two-page resume to present yourself as a “possible and strong candidate.” Use the pyramid to guide your way. If it’s been a while since you last updated your resume, for creative ideas, be sure to check out the new resume templates on Microsoft Word or a similar editing tool. Give it a try, please.
Next, connect your resume with your digital presence on LinkedIn. Download the LinkedIn mobile app. In a few easy steps, you can create your digital signature with a QR code and save it with your photos.
Smart tip: When a recruiter scans your “Unique QR code,” they can see your accomplishments and connect through LinkedIn Messaging. Super cool!
We live in a digital world, yet the resume is still a PDF or even a paper copy. No problem! Simply, add your QR code to your resume.
Invest in yourself, keep learning, and add your skills to your LinkedIn profile. Show the world skills that make you a dream hire for any organization!
Do all these things, and, yes, soon, you will have that email or in-mail message from a recruiter asking to call you for an interview. That next opportunity is just around the corner. Believe in yourself.
Best of Luck!
Jyotsna
Jyotsna Manikantan is a Lead Product Manager, Portfolio Strategy & Operations, and is celebrating her 13th-year @ADP
Does culture really eat strategy for breakfast?
https://eng.lifion.com/yes-culture-does-eat-strategy-for-breakfast-638ae19fc506
Yes, Culture DOES Eat Strategy for Breakfast
“The best employees love what they do”, former PepsiCo CEO Indra Nooyi once responding to a crowd of bright-eyed summer IT interns. Quite a simple answer to the question “what do the best employees do, and how can I be like them?”. The more you think about the fast response Indra had given, the more you realize it extends to more than simply ‘what’ we do. As one of those interns, I didn’t quite understand the depth of this statement until I had exposure to different roles later in my career.
Loving what you do is inherently important, yes, but it’s not exactly the single most important aspect to making you an incredible employee. Whether people want to acknowledge it or not, what you do for work becomes part of you. For example, I may have been called a ‘tech bro’ once or twice after being asked what industry I’m in, to which I usually respond: “I’m not fixing your computer”. After the laughter calms down, I usually begin talking about the work I do, and then inevitably, the “are you happy?” question arises. Most people respond fairly quickly, and to varying degrees, but if you stop and think about the rounded picture of happiness at your job, it usually comes down to one thing: personal fulfillment based on the investment of your time. Many concepts can be abstracted from this — work relationships/enjoyment, culture-driven benefits, monetary compensation, societal impact, etc. How many start-ups have you heard say that they’re going to change the world? Have you ever stopped to think about this as a marketing ploy, rather than a vision statement?
We all have different opinions on what’s heavier when it comes to our personal happiness, but I’ve found that one particularly weighs more than others, and that’s culture. From an organizational perspective, culture is tough to get right and easy to ruin, but when fostered correctly can truly breed the best and happiest employees. Culture is something that’s created organically, targeting the basic human desire to belong, fit in, and feel like a contributor to the group. A group that allows for the dynamism of thought, and freedom to express it without judgment, enables each individual to feel heard while allowing the best outcome to become possible. This, in turn, opens up avenues for new and personal discussions between individuals, potentially turning into friendships. When an employee goes into work every morning, considering his/her coworkers to be people he/she looks forward to working with, that’s where the magic happens, and culture is born. Good relationships grow general satisfaction with a person’s environment. Happy employees are much more likely to go out of their way to make the organization succeed, which full circle, makes them a good employee in the eyes of the organization.
How often in pop culture is work-life balance portrayed with someone comically saying “I don’t know any of you outside of work”. Now, full disclosure, I’ve never once heard anyone explicitly say this, but have seen individuals act in this way. This mentality simply doesn’t instill a sense of trust, no matter what organization or industry you work in. Don’t get me wrong, separation is good and healthy to have, but communication is key.
At Lifion, by ADP, we’re in the business of ‘Human Capital Management’, and we understand that if an organization doesn’t manage its resources correctly, the strongest culture won’t prevail. Everyone wants to be treated as an individual, rather than just another employee. No inhuman litmus test for happiness should ever suffice for a one on one, a drink together at happy hour, or a sincere “hey, how are you doing?”.
In summary, you never know the amount of impact you have on someone’s daily work experience. People are more than just resources to get a job done and respond well when treated as an individual. At the end of the day, you can have the best strategy in the world, but with no one to build it, it’s not worth anything. Simply put, whether you are a company, a manager, or a coworker, the best advice I can give: be human.
Creating human-focused solutions in today’s product strategy
ADP Business Anthropologist Martha Bird sat down with Daniel Litwin, the Voice of B2B, at CES 2020, discussing a wide range of topics related to how her anthropological work and research impacts businesses and consumer needs.
Bird has worked for numerous companies in the field of business anthropology since the early 2000s, working to create human-focused solutions to business needs.
Bird and Litwin touch on their CES experience, a modern focus on human-centered and human-responsive products and how those concepts affect consumer product development, consumer longing for personalized experiences, and more.
Let’s Talk About Sets
https://eng.lifion.com/lets-talk-about-sets-813dfeb2185
Let’s Talk About Sets
A re-introduction to JavaScript Sets and the new Set methods
Let’s talk about you and me and how we used to find unique items before ES6. We really only had two ways to do it (if you had another one let me know). On the first one, we would create a new emtpy object, iterate through the items we wanted to deduplicate, we would create a new property using the item as the key and something like “true” as the value, then we would get the list of keys of that new object and we were done. In the second way, we would create a new empty array, iterate through the items, and for each item, check if the item existed in the array, if it was already there, continue, if not, add it. By the end the array would contain all the unique items.
ES6 introduced Sets, a new data structure with a very simple API to handle unique items that is not just convenient but also very fast. The intention of this article is to introduce you to some new methods coming to Sets soon that will make them even more useful, but before, let’s remember the basics.
Here in Lifion we are big users of JavaScript, about 90% of our platform services are Node.js-based. If you are interested to see some examples of how Sets are used in our codebase, check our open source projects in Lifion’s GitHub profile.
The basics of Sets
To create a new set we only need to use the constructor. We can optionally pass any iterator, such as an array or a string, and the iterated items will become elements of the new set (repeated items will be ignored).
const emptySet = new Set();
const prefilledSet = new Set(['
Lifion at ADP’s cloud transformation journey
https://eng.lifion.com/lifions-cloud-transformation-journey-2333b7c0897d
Lifion’s Cloud Transformation Journey
On moving to managed services in a microservice architecture
Since Lifion’s inception as ADP’s next-generation Human Capital Management (HCM) platform, we’ve made an effort to embrace relevant technology trends and advancements. From microservices and container orchestration frameworks to distributed databases, and everything in between, we’re continually exploring ways we can evolve our architecture. Our readiness to evaluate non-traditional, cutting edge technology has meant that some bets have stuck whereas others have pivoted.
One of our biggest pivots has been a shift from self-managed databases & streaming systems, running on cloud compute services (like Amazon EC2) and deployed with tools like Terraform and Ansible, towards fully cloud-managed services.
When we launched the effort to make this shift in early 2018, we began by executing a structured, planned initiative across an organization of 200+ engineers. After overcoming the initial inertia, the effort continued to gain momentum, eventually taking a life of its own, and finally becoming fully embedded in how our teams work.
Along the way, we’ve been thinking about what we can give back. For example, we’ve previously written about a node.js client for AWS Kinesis that we’re working on as an open source initiative.
AWS’s re:Invent conference is perhaps the largest global cloud community conference in the world. In late 2018, we presented our cloud transformation journey at re:Invent. As you can see in the recording, we described our journey and key learnings in adopting specific AWS managed services.
In this post, we discuss key factors that made the initiative successful, its benefits in our microservice architecture, and how managed services helped us shift our teams’ focus to our core product while improving overall reliability.
Why Services Don’t Share Databases
The notion of services sharing databases, making direct connections to the same database system and being dependent on shared schemas, is a recognized micro-service anti-pattern. With shared databases, changes in the underlying database (including schemas, scaling operations such as sharding, or even migrating to a better database) become very difficult with coordination required between multiple service teams and releases.
As Amazon.com CTO Werner Vogels writes in his blog:
Each service encapsulates its own data and presents a hardened API for others to use. Most importantly, direct database access to the data from outside its respective service is not allowed. This architectural pattern was a response to the scaling challenges that had challenged Amazon.com through its first 5 years…
And Martin Fowler on integration databases:
On the whole integration databases lead to serious problems becaue [sic] the database becomes a point of coupling between the applications that access it. This is usually a deep coupling that significantly increases the risk involved in changing those applications and making it harder to evolve them. As a result most software architects that I respect take the view that integration databases should be avoided.
The Right Tool for the Job
Applying the database per service principal means that, in practice, service teams have significant autonomy in selecting the right database technologies for their purposes. Among other factors, their data modeling, query flexibility, consistency, latency, and throughput requirements will dictate technologies that work best for them.
Up to this point, all is well — every service has isolated its data. However, when architecting a product with double digit domains, several important database infrastructure decisions need to be made:
- Shared vs dedicated clusters: Should services share database clusters with logically isolated namespaces (like logical databases in MySQL), or should each have its own expensive cluster with dedicated resources?
- Ownership: What level of ownership does a service team take for the deployment, monitoring, reliability, and maintenance of their infrastructure?
- Consolidation: Is there an agreed set of technologies that teams can pick from, is there a process for introducing something new, or can a team pick anything they like?
From Self-Managed to Fully Managed Services
When we first started building out our services, we had a sprawl of supporting databases, streaming, and queuing systems. Each of these technologies was deployed on AWS EC2, and we were responsible for the full scope of managing this infrastructure: from the OS level, to topology design, configuration, upgrades and backups.
It didn’t take us long to realize how much time we were spending on managing all of this infrastructure. When we made the bet on managed services, several of the decisions we’d been struggling with started falling into place:
- Shared vs dedicated clusters: Dedicated clusters for services, clearly preferable from a reliability and availability perspective, became easier to deploy and maintain. Offerings like SQS, DynamoDB, and Kinesis with no nodes or clusters to manage removed the concern altogether.
- Ownership: Infrastructure simplification meant that service teams were able to develop further insight into their production usages, and take greater responsibility for their infrastructure.
- Consolidation: We were now working with a major cloud provider’s service offerings, and found that there was enough breadth to span our use cases.
Evolutionary Architecture
On our Lifion engineering blog, we’ve previously written about our Lifion Developer Platform Credos. One of these speaks to the evolutionary nature of our work:
- Build to evolve: We design our domains and services fully expecting that they will evolve over time.
- Backwards compatible, versioned: Instead of big bang releases, we use versions or feature flags letting service teams deploy at any time without coordinating dependencies.
- Managed deprecations: When deprecating APIs or features, we carefully plan the impact and ensure that consumer impact is minimal.
When we started adopting managed services, we went for drop-in replacements first (for example, Aurora MySQL is wire compatible with the previous MySQL cluster we were using). This approach helped us to get some early momentum while uncovering dimensions like authentication, monitoring, and discoverability that would help us later.
Our evolutionary architecture credo helped to ensure that the transition would be smooth for our services and our customers. Each deployment was done as a fully online operation, without customer impact. We recognize that we will undergo more evolutions, for which we intend to follow the same principles.
Performance implications of misunderstanding Node.js promises
https://eng.lifion.com/promise-allpocalypse-cfb6741298a7
Promise.allpocalypse
The performance implications of misunderstanding Node.js promises
There is nothing quite like the existential dread of realizing you may have a performance issue. Months, possibly years, worth of codebase are suddenly now failing you in a live environment. Reproduction is difficult, and even if you can reproduce it on your dev machine your usual methods of debugging are likely useless. Sure, you can litter your code with entirely too much analytics, or leverage Node’s own profiling, but this results in a data hose that takes a deep understanding of Node, V8, and data science itself to make sense of it. This is a rather dire situation.
Enter Clinic.js; a tool from NearForm that utilizes Node and V8’s own internal profiling, as well as inserting probes into the JavaScript itself, to analyze and produce easy to consume diagnostic information as well as recommend next steps. The power and capabilities of Clinic.js can be seen in great detail on its website, though as the title implies, I am going to employ Clinic.js to focus on the danger of misusing Promise.all().
A common scenario in any service is you may have some asynchronous operation that you need to perform on a data set of unknown length. Your first instinct may look something like this:
You quickly discover that this is not very fast. Simple interrogation of CPU utilization will show Node is spending a lot of time waiting. Depending on your lint rules, you may not even be able to get this code past a CI build. The solution for many is to simply fire all the promises then await all of them. That, in and of itself, is not a bad thing; for example, say you know that every time you execute a given route, you need to make 3 unrelated calls to other services. Since you know it will always create 3 promises, this is bounded and safe. The problems arise when the promise creation is potentially unbounded.
You test your code and discover it is much faster, Node is not spending time waiting around, and the lint error is gone. You merge and deploy your code, all the integration tests pass, and you go home to watch The Expanse. Suddenly, right when the Rocinante (a legitimate salvage) is entering a tense battle, you start getting PagerDuty alerts. Your services containers are getting restarted constantly due to Out of Memory exceptions, the container host is experiencing socket exhaustion, and your response times are through the roof. You have a major performance and scalability issue.
Benchmarking unbounded promise scenarios
This is a nice clean scenario, so the suspect code is easily identified, but what if it isn’t? Why does this code fail to scale? This is where Clinic.js shines. For the sake of exercising Clinic.js, I created 6 scenarios that all perform the same asynchronous work some n times using different strategies. They are using for…of, the new ES2018 for await…of, the Promise.all() scenario described above, using Bluebird.map() with a concurrency limit set, the promise-limit module, and the p-limit module.
For the test, I start a clustered web server that accepts a get and post request. Both verbs will perform some arbitrary work before completing the request. In the test itself, I get the data, do arbitrary work, and post the results back. In both cases the work is primarily parsing JSON and manipulating the object in an attempt to simulate a common Node workload. Each test run does this work 500 times, for each element in a mock collection. For the limiting modules, they are all set to allow 10 concurrent promises. All tests were performed using Node v12.14.0. The average execution times over 10 test runs can be seen below.
╔═══════════════╦══════════════════════════════════╗
║ Test ║ Average Execution Time (Seconds) ║
╠═══════════════╬══════════════════════════════════╣
║ await-for-of ║ 6.943 ║
║ bluebird-map ║ 4.550 ║
║ for-of ║ 6.745 ║
║ p-limit ║ 4.523 ║
║ promise-all ║ 4.524 ║
║ promise-limit ║ 4.457 ║
╚═══════════════╩══════════════════════════════════╝
The results here certainly reflect the performance improvement when going from for…of to issuing concurrent promises, but the limiting libraries have similar execution times. Let’s dive deeper into what is going on.
for…of and for await…of
In the for await…of and for..of scenarios (respectively), the CPU utilization is bouncing around and is averaging well below 100%. This means the Node process is spending a lot of time waiting around. The execution time is also non-ideal at roughly 7 seconds averaged over 10 runs. Clinic detects this and reports a potential I/O wait issue bottlenecking the process. It recommends running a ‘bubbleprof’ to better identify the issue.
Note the light blue segment in the graph, this represents network I/O waits. It is spending quite a bit of time waiting for network operations. Perhaps we can speed this up by issuing them concurrently.
Promise.all()
In the Promise.all() scenario, execution time has improved dramatically to 4.5 seconds on average. The CPU is now being efficiently utilized. However, notice the memory usage spiking up much higher with incredibly high event loop delay. Clinic detects this issue and reports a potential event loop issue. Why is this?
It is a common misunderstanding that promises are concurrent. While there is some threading in V8, JavaScript itself is executed synchronously. When a new promise is created, it is eagerly executed until it reaches a truly asynchronous operation or the end of the JavaScript code path. Consider the below example.
As you can see, the promise callback is invoked first, then it executes to the end of the block. Only after that do the `then` blocks fire. This, however, is only part of the story.
In the case of the Promise.all() code above, it will create a request for every item in the collection before any `then` block is resolved. This is true even if the early requests complete before it reaches the end of the collection. It will not process the responses until the event loop ticks over. This is due to I/O callbacks not being invoked until the event loop ticks into its I/O phase. It is only at this point `.then()`, or the code block proceeding `await`, is executed.
Ultimately the creation of an unbounded number of promises will result in all of the response objects becoming queued in memory and waiting for the event loop to tick over. This can swamp the Node process with excessive memory allocations, or worse, cause out of memory events. When these objects go out of scope, it will freeze up when the garbage collector executes as it is primarily a blocking, O(N) operation. It also means when the event loop finally does tick over, all of the `.then()` callbacks of completed requests fire synchronously. If this contains heavy work, the event loop will be stuck in the callback phase until this work is completed or another truly asynchronous operation is encountered. This means any events, inbound requests, timers, etc. will not be handled until the ‘.then()’ callbacks are all completed. The biggest concern here is that response times can spike to several seconds. To solve this, we need to limit the number of concurrent promises created.
Limiting simultaneous promises
As indicated in the graphs, Bluebird.map() with a concurrency limit pretty much eliminates the event loop blocking and the high memory usage while maintaining efficient CPU utilization. It matches the performance of Promise.all(), completing the test in 4.5s. Bluebird however has lost its performance edge in recent versions of V8 when compared to native promises and is a rather heavy complete Promise replacement. Let’s consider some other native promise options.
The promise-limit module also resolves our issue, and against matches the performance of Promise.all() at 4.5s. This is due to the aforementioned pressure on Node’s memory system creating unnecessary memory allocations and complex garbage collection events. The overhead of this module does get flagged by Clinic.js as creating some event loop blocking. It notably also is nowhere near as popular as the last module, p-limit.
The p-limit module has the least amount of event loop blocking of all the options, while providing the same performance as Promise.all() at 4.5s. Given the popularity compared to promise-limit and the performance of this module, this is the clear winner.
Conclusion
Clinic.js enables introspection into how your Node application is performing beyond simple top line execution times. It introspects not only your resource utilization, but what the code itself is doing. By no means have I explored all the capabilities of this tool here, however it grants us insight into why a specific codepath may appear to perform without issue on its own, but wreak havoc inside of a live service. NearForm presented this to us during Node Day 2019 and it has quickly become an indispensable debugging tool for us. Its ease of use and clear presentation helps not only quickly identify problematic code, but also convey the “why” to others who are not deep into the debugging process with you. It was the obvious choice to illustrate the issues with unbounded promise creation.
Promises and the async/await syntax lead to far clearer, more maintainable JavaScript code. Its appearance of parallelism is an easy trap to fall into, especially for polyglot developers who are used to similar features in other languages. While unbounded thread creation has its own problems, unbounded Promise creation will bog down an entire service, affecting all code paths in the application, and will cause out of memory events as you scale. Usage of modules that allow you to limit concurrent promises can provide even better absolute performance than Promise.all() and enable your application to scale better. It’s this engineer’s perspective that Promise.all() should be banished from your code in all scenarios where you cannot guarantee at the time of coding how many promises will be created.