Close up of lights on computer devices in server room

How to select, gather, clean, and test data for machine learning systems

August 4, 2020/in Engineering, Women in STEM Alpharetta, EJD, Hyderabad cleaning and testing, cleaning and testing data, data for machine-learning, data for machine-learning systems, datacloud team, gathering cleaning, gathering cleaning and testing, gathering cleaning and testing data, information, job titles, machine learning, machine-learning systems, personal information, right data, selecting gathering, selecting gathering cleaning, selecting gathering cleaning and testing, testing data, testing data for machine-learning, testing data for machine-learning systems /by myto

https://explore.adp.com/spark3/how-data-becomes-insight-the-right-data-matters-454FC-31577B.html

How Data Becomes Insight:

The Right Data Matters

By SPARK Team

What goes into selecting, gathering, cleaning and testing data for machine-learning systems?

It’s not enough to have a lot of data and some good ideas. The quality, quantity and nature of the data is the foundation for using it effectively.

We asked members of the ADP^® DataCloud Team to help us understand what goes into selecting, gathering, cleaning and testing data for machine-learning systems.

Q: How do you go from lots of information to usable data in a machine-learning system?

DataCloud Team: The first thing to figure out is whether you have the information you want to answer the questions or solve the problem you’re working on. So, we look at what data we have and figure out what we can do with it. Sometimes, we know right away we need some other data to fill in gaps or provide more context. Other times, we realize that some other data would be useful as we build and test the system. One of the exciting things about machine learning is that it often gives us better questions, which sometimes need new data that we hadn’t thought about when we started.

Once you know what data you want to start with, then you want it “clean and normalized.” This just means that the data is all in a consistent format so it can be combined with other data and analyzed. It’s the process where we make sure we have the right data, get rid of irrelevant or corrupt data, that the data is accurate and that we can use it with all our other data when the information is coming from multiple sources.

A great example is job titles. Every company uses different titles. A “director” could be an entry-level position, a senior executive, or something in between. So, we could not compare jobs based on job titles. We had to figure out what each job actually was and where it fit in a standard hierarchy before we could use the data in our system.

Q: This sounds difficult.

DataCloud Team: There’s a joke that data scientists spend 80 percent of their time cleaning data and the other 20 percent complaining about it.

At ADP, we are fortunate that much of the data we work with is collected in an organized and usable way through our payroll and HR systems, which makes part of the process easier. Every time we change one of our products or build new ones, data compatibility is an important consideration. This allows us to work on the more complex issues, like coming up with a workable taxonomy for jobs with different titles.

But getting the data right is foundational to everything that happens, so it’s effort well spent.

Q: If you are working with HR and payroll data, doesn’t it have a lot of personal information about people? How do you handle privacy and confidentiality issues?

DataCloud Team: We are extremely sensitive to people’s privacy and go to great lengths to protect both the security of the data we have as well as people’s personal information.

With machine learning we are looking for patterns, connections or matches and correlations. So, we don’t need personally identifying data about individuals. We anonymize the information and label and organize it by categories such as job, level in hierarchy, location, industry, size of organization, and tenure. This is sometimes called “chunking.” For example, instead of keeping track of exact salaries, we combine them into salary ranges. This both makes the information easier to sort and protects people’s privacy.

With benchmarking analytics, if any data set is too small to make anonymous ― meaning it would be too easy to figure out who it was ― then we don’t include that data in the benchmark analysis.

Q: Once you have your initial data set, how do you know when you need or want more?

DataCloud Team: The essence of machine learning is more data.

We want to be able to see what is happening over time, what is changing, and be able to adjust our systems based on this fresh flow of data. As people use the programs, we are also able to validate or correct information. For example with our jobs information, users tell us how the positions in their organization fit into our categories. This makes the program useful to them, and makes the overall database more accurate.

As people use machine-learning systems, they create new data which the system learns from and adjusts to. It allows us to detect changes, see cycles over time, and come up with new questions and applications. Sometimes we decide we need to add a new category of information or ask the system to process the information a different way.

These are the things that both keep us up at night and make it exciting to show up at work every day.

Learn more by getting our guide, “Proving the Power of People Data.”

« All Blogs

Person gesturing toward large computer screen

Performance implications of misunderstanding Node.js promises

August 4, 2020/in Engineering, Voice of Our People Alpharetta, NYC bluebird.map with concurrency limit, cause out of memory events, chain execution order example, clinic.js doctor, clinic.js doctor output, code, concurrency limit, concurrent promises, doctor output, event loop, event loop blocking, matches the performance of promise.all, module test code, output for for await…, p-limit module, promise-limit module, spending a lot of time, test code, time waiting, unbounded promise creation /by myto

https://eng.lifion.com/promise-allpocalypse-cfb6741298a7

Promise.allpocalypse

The performance implications of misunderstanding Node.js promises

Ali Yousuf

Jan 22 · 8 min read

There is nothing quite like the existential dread of realizing you may have a performance issue. Months, possibly years, worth of codebase are suddenly now failing you in a live environment. Reproduction is difficult, and even if you can reproduce it on your dev machine your usual methods of debugging are likely useless. Sure, you can litter your code with entirely too much analytics, or leverage Node’s own profiling, but this results in a data hose that takes a deep understanding of Node, V8, and data science itself to make sense of it. This is a rather dire situation.

Enter Clinic.js; a tool from NearForm that utilizes Node and V8’s own internal profiling, as well as inserting probes into the JavaScript itself, to analyze and produce easy to consume diagnostic information as well as recommend next steps. The power and capabilities of Clinic.js can be seen in great detail on its website, though as the title implies, I am going to employ Clinic.js to focus on the danger of misusing Promise.all().

A common scenario in any service is you may have some asynchronous operation that you need to perform on a data set of unknown length. Your first instinct may look something like this:

for…of over unknown collection with await in loop

You quickly discover that this is not very fast. Simple interrogation of CPU utilization will show Node is spending a lot of time waiting. Depending on your lint rules, you may not even be able to get this code past a CI build. The solution for many is to simply fire all the promises then await all of them. That, in and of itself, is not a bad thing; for example, say you know that every time you execute a given route, you need to make 3 unrelated calls to other services. Since you know it will always create 3 promises, this is bounded and safe. The problems arise when the promise creation is potentially unbounded.

Promise.all() on an entire unknown collection

You test your code and discover it is much faster, Node is not spending time waiting around, and the lint error is gone. You merge and deploy your code, all the integration tests pass, and you go home to watch The Expanse. Suddenly, right when the Rocinante (a legitimate salvage) is entering a tense battle, you start getting PagerDuty alerts. Your services containers are getting restarted constantly due to Out of Memory exceptions, the container host is experiencing socket exhaustion, and your response times are through the roof. You have a major performance and scalability issue.

Benchmarking unbounded promise scenarios

This is a nice clean scenario, so the suspect code is easily identified, but what if it isn’t? Why does this code fail to scale? This is where Clinic.js shines. For the sake of exercising Clinic.js, I created 6 scenarios that all perform the same asynchronous work some n times using different strategies. They are using for…of, the new ES2018 for await…of, the Promise.all() scenario described above, using Bluebird.map() with a concurrency limit set, the promise-limit module, and the p-limit module.

For the test, I start a clustered web server that accepts a get and post request. Both verbs will perform some arbitrary work before completing the request. In the test itself, I get the data, do arbitrary work, and post the results back. In both cases the work is primarily parsing JSON and manipulating the object in an attempt to simulate a common Node workload. Each test run does this work 500 times, for each element in a mock collection. For the limiting modules, they are all set to allow 10 concurrent promises. All tests were performed using Node v12.14.0. The average execution times over 10 test runs can be seen below.

╔═══════════════╦══════════════════════════════════╗
║     Test      ║ Average Execution Time (Seconds) ║
╠═══════════════╬══════════════════════════════════╣
║ await-for-of  ║                            6.943 ║
║ bluebird-map  ║                            4.550 ║
║ for-of        ║                            6.745 ║
║ p-limit       ║                            4.523 ║
║ promise-all   ║                            4.524 ║
║ promise-limit ║                            4.457 ║
╚═══════════════╩══════════════════════════════════╝

The results here certainly reflect the performance improvement when going from for…of to issuing concurrent promises, but the limiting libraries have similar execution times. Let’s dive deeper into what is going on.

for…of and for await…of

for…of test code

for await…of test code

In the for await…of and for..of scenarios (respectively), the CPU utilization is bouncing around and is averaging well below 100%. This means the Node process is spending a lot of time waiting around. The execution time is also non-ideal at roughly 7 seconds averaged over 10 runs. Clinic detects this and reports a potential I/O wait issue bottlenecking the process. It recommends running a ‘bubbleprof’ to better identify the issue.

Note the light blue segment in the graph, this represents network I/O waits. It is spending quite a bit of time waiting for network operations. Perhaps we can speed this up by issuing them concurrently.

Promise.all()

Promise.all() test code

In the Promise.all() scenario, execution time has improved dramatically to 4.5 seconds on average. The CPU is now being efficiently utilized. However, notice the memory usage spiking up much higher with incredibly high event loop delay. Clinic detects this issue and reports a potential event loop issue. Why is this?

It is a common misunderstanding that promises are concurrent. While there is some threading in V8, JavaScript itself is executed synchronously. When a new promise is created, it is eagerly executed until it reaches a truly asynchronous operation or the end of the JavaScript code path. Consider the below example.

Promise chain execution order example

As you can see, the promise callback is invoked first, then it executes to the end of the block. Only after that do the `then` blocks fire. This, however, is only part of the story.

Async chain execution order example

In the case of the Promise.all() code above, it will create a request for every item in the collection before any `then` block is resolved. This is true even if the early requests complete before it reaches the end of the collection. It will not process the responses until the event loop ticks over. This is due to I/O callbacks not being invoked until the event loop ticks into its I/O phase. It is only at this point `.then()`, or the code block proceeding `await`, is executed.

Ultimately the creation of an unbounded number of promises will result in all of the response objects becoming queued in memory and waiting for the event loop to tick over. This can swamp the Node process with excessive memory allocations, or worse, cause out of memory events. When these objects go out of scope, it will freeze up when the garbage collector executes as it is primarily a blocking, O(N) operation. It also means when the event loop finally does tick over, all of the `.then()` callbacks of completed requests fire synchronously. If this contains heavy work, the event loop will be stuck in the callback phase until this work is completed or another truly asynchronous operation is encountered. This means any events, inbound requests, timers, etc. will not be handled until the ‘.then()’ callbacks are all completed. The biggest concern here is that response times can spike to several seconds. To solve this, we need to limit the number of concurrent promises created.

Limiting simultaneous promises

Bluebird.map() with concurrency limit test code

As indicated in the graphs, Bluebird.map() with a concurrency limit pretty much eliminates the event loop blocking and the high memory usage while maintaining efficient CPU utilization. It matches the performance of Promise.all(), completing the test in 4.5s. Bluebird however has lost its performance edge in recent versions of V8 when compared to native promises and is a rather heavy complete Promise replacement. Let’s consider some other native promise options.

promise-limit module test code

The promise-limit module also resolves our issue, and against matches the performance of Promise.all() at 4.5s. This is due to the aforementioned pressure on Node’s memory system creating unnecessary memory allocations and complex garbage collection events. The overhead of this module does get flagged by Clinic.js as creating some event loop blocking. It notably also is nowhere near as popular as the last module, p-limit.

p-limit module test code

The p-limit module has the least amount of event loop blocking of all the options, while providing the same performance as Promise.all() at 4.5s. Given the popularity compared to promise-limit and the performance of this module, this is the clear winner.

Conclusion

Clinic.js enables introspection into how your Node application is performing beyond simple top line execution times. It introspects not only your resource utilization, but what the code itself is doing. By no means have I explored all the capabilities of this tool here, however it grants us insight into why a specific codepath may appear to perform without issue on its own, but wreak havoc inside of a live service. NearForm presented this to us during Node Day 2019 and it has quickly become an indispensable debugging tool for us. Its ease of use and clear presentation helps not only quickly identify problematic code, but also convey the “why” to others who are not deep into the debugging process with you. It was the obvious choice to illustrate the issues with unbounded promise creation.

Promises and the async/await syntax lead to far clearer, more maintainable JavaScript code. Its appearance of parallelism is an easy trap to fall into, especially for polyglot developers who are used to similar features in other languages. While unbounded thread creation has its own problems, unbounded Promise creation will bog down an entire service, affecting all code paths in the application, and will cause out of memory events as you scale. Usage of modules that allow you to limit concurrent promises can provide even better absolute performance than Promise.all() and enable your application to scale better. It’s this engineer’s perspective that Promise.all() should be banished from your code in all scenarios where you cannot guarantee at the time of coding how many promises will be created.

« All Blogs

Diverse group of multicultural ADP employees

We All Want to Belong at Work

August 4, 2020/in Culture, Diversity & Inclusion, Diversity, Equity & Inclusion, Impact, Impact & Innovation Alpharetta, Roseland adp's business, adp's business anthropologist, adp's business anthropologist martha, adp's business anthropologist martha bird, anthropologist martha bird, belong at work, belonging, belonging at work, business anthropologist, business anthropologist martha, business anthropologist martha bird, commonalities and accepting, culture of belonging, finding commonalities, finding commonalities and accepting, key to belonging, people, sense of belonging, social network, work /by myto

https://www.adp.com/spark/articles/2019/10/we-all-want-to-belong-at-work.aspx

We All Want to Belong at Work

by Heather Bussing

“Finding commonalities and accepting differences is the key to belonging,” said ADP’s business anthropologist, Martha Bird.

When I started to consider belonging at work, I knew exactly who to call — ADP’s business anthropologist, Martha Bird [MB]. Here’s some of our conversation about why belonging matters and what organizations can do to create and sustain a culture of belonging.

HB: Having a sense of belonging seems so important to how we move through the world and how we relate to our work. What is belonging?

MB: Belonging almost strikes people as poetic. It seems like a feeling, so it can resist the critical lens we need to unpack it.

People think of belonging as a psychological state, but it is actually cultural. It’s the notion of being inside or outside and relates to enacted phenomena like what the cultural norms are around us and how we compare ourselves to those norms.

Everything cultural is nested in other things and is influenced by power, resources, how things have been done in the past, and what the expectations are for the people involved.

Kids can feel like they don’t belong because of their clothes. New employees can feel like they don’t belong because of the jargon used in the organization. I’m a social scientist surrounded by tech people and it’s not surprising that my sense of belonging is tested from time to time. Ultimately, I’m privileged to feel I’m part of something bigger than myself.

HB: What makes a culture of belonging? It seems like belonging is relational. It’s partly how I perceive the circumstances and culture, how people already in that culture see it, and what’s actually going on regardless of our individual perceptions and opinions.

MB: There are so many ways to feel like you don’t belong — socially, economically, intellectually, emotionally. It’s that sense of other. To make sense of it, we can look at othering, break it down, and pick it apart to see what’s happening. We identify the discreet instances where someone feels alienated and read the cultural cues about what is happening. This gives us information about the culture.

There is no universal recipe for what makes a healthy culture. There are many good and right ways to do things.

It partly has to do with a culture’s view of the individual and how the individual should relate to others. In the United States, belonging often evokes family, but we also have strong cultural values in individuality and being recognized and valued as an individual. In other cultures, a sense of harmony is highly valued and working toward common goals is more important than individual achievement.

A culture of belonging fundamentally has to do with common goals and values, respect for each other, and a sense of our shared humanity.

HB: How can we help people feel like they belong at work?

MB: We want workplaces where people feel like they can be themselves, but are also working with others to do the work. It’s less about fitting in and more about complementing. There has to be room for difference. It’s like an orchestra where the manager is the conductor and we have all these different instruments playing different parts in the same piece of music. We don’t want just the violins or the tubas. We need all the different sounds, rhythms and harmonies.

Belonging at work starts with leaders modeling the values and behavior for their teams. Is it comfortable to embody those values? Sometimes that means being vulnerable and asking for help.

I recently gave a big speech to a large group of people in a setting where I felt anxious. Walking up to the stage, I decided to tell the audience that and ask for help. So I explained how I was feeling and asked them to tell me, “It’s okay, Martha!” It was great, so I asked them to do it again. And they did! I felt so much better and they were all on my team at that point, because I was vulnerable and asked them to help me in a way they could.

In cultures of belonging, it’s okay to be honest about what’s going on, even if it’s that you don’t feel included.

HB: What are some specific things that managers or leaders can do to foster belonging at work?

MB: At the organizational level, it’s essential to ensure that the values of the organization exist at every level and in every manager without exception. It’s also important to consider how to structure teams and make sure they can communicate effectively, based on where and how people work.

At the team level, good manager training is key. Managers need skills in working with teams, allowing for different views, and figuring out how to handle disagreements and how decisions get made. When people can weigh in on something, there is a sense of being in it together.

It’s important to see each other as people, not work roles. Connecting in person and outside of work makes a difference. We need to tell and know each other’s stories and create opportunities for sharing. Have lunch, have informal video meetings where everyone gets to tell something about themselves. I was in a meeting recently where we all told the story of our names. I learned a lot and felt like the people who heard my story knew me a little better, too.

We need more awareness and cultural consciousness by design. People are fundamentally creative and want to learn. We all have different experiences and different lives.

Finding commonalities and accepting those differences is the key to belonging.

« All Blogs

Are you in the right job? Finding your passion and advancing your career.

April 9, 2020/in Career Advice & Insights, Career Insights, Voice of Our People, Women in STEM Alpharetta, Hyderabad, NYC adp, balance, career, experience, job, jsf and dojo, money the people, new project, people, project, right, right job, team, technologies, technologies the money, technologies the money the people, web, work, work-life, work-life balance /by myto

Best advice Samantha ever received? “If you are the smartest person in the room, you’re in the wrong room.”

Samantha Ortiz started out as a zookeeper. Then she went into marketing. Now she’s a software engineer at ADP’s Innovation Lab working on its NextGen Platform. Her ever-present curiosity, creativity, and passion for understanding behavior and solving problems have been common threads throughout her varied life experiences.

From Horses to Manatees …

Samantha was born and raised in the Bronx and was comfortable in the city, but she always had an interest in the natural world around her. Her older brother raised tropical poison dart frogs at home, and she was mesmerized as a child while observing them in their terrariums. She also spent most of her free time riding and training horses at Riverdale Riding Center in the Bronx’s Van Cortlandt Park, expanding the interest she had in animals.

After spending most of her life in the city, her family moved to Florida, and she was suddenly surrounded by agriculture. She made the most of her new journey in life. Sam joined the local Future Farmers of America, becoming the chapter’s student advisor and showing a pig at the county fair. Sam also continued her love of riding horses by finally fulfilling her dream of owning a horse and practicing dressage. “My mother was always so supportive of my adventures with animals,” she says. “Even if she felt a little out of place on the farm – she is a New York native, after all – she was there for me every step I took.”

Her passion for understanding living things continued in college as she studied psychology at New College of Florida, focusing on animal behavior and conservation. She spent her undergraduate years working with a wide range of animals, from studying manatee sensory behavior to handedness preferences in lemurs. Sam also conducted research on numerous species of fish, including Indian Mudskippers (an amphibious fish), and Stoplight Parrotfish. She spent several summers in Panama working on her thesis research on Parrotfish’s feeding behavior and its effects on coral reefs. She was also introduced to design, exploring zoo and aquarium design, its effects on animals, and how it fosters conservation behaviors in visitors.

… and from Reptiles to Coding

After college, she worked as a zookeeper in Florida, caring for animals and presenting reptile educational programs to visitors. “So many people were curious about alligator behavior, especially since we were in Florida,” Samantha says. “I shared with them how human actions, particularly humans feeding wildlife, would contribute negatively to the animals’ natural behavior, making them more dangerous as they’d become acquainted to people. Everyone’s actions and behaviors affect something or someone else.”

Samantha’s natural interest in behavior took her down a winding path beyond the natural world and into technology.

A relationship took her to Texas, where she worked for a digital marketing consultancy. While she worked as a copywriter and copy editor, Sam also started to combine her knowledge of behavior with the tools of technology. The marketing campaigns she ran combined multiple applications, and she realized she wanted a deeper understanding of the software she was using. How did it target specific demographics? What data did it use to determine which campaigns would trigger actionable behavior in users? “Where I went to college, they encourage you to be an independent thinker and deeply analyze things,” she says. “That’s how I have always approached everything.”

Samantha started to explore coding by teaching herself web development through an online program. After relocating back to New York, she applied to Hack Reactor, an intensive, full-time coding boot camp

“At first, I was telling myself I’m not ready, maybe it’s too late, I don’t know if I can do this,” she remembers. “So, I started with some prep classes before deciding that software engineering was what I wanted to do.”

Coming to ADP

While Samantha was at Hack Reactor, she built applications with classmates, and two of them went on to work at ADP. She was invited to a networking night at the ADP Innovation Lab, where “I met lots of fun, intelligent people who love what they do. I started talking to Yaara (Katz) and we just clicked. It was so great to meet another female software engineer with a passion for her work. We laughed and I really felt comfortable. I knew I was home.”

Samantha loves software engineering, noting, “It’s such a creative process. I have loved writing my whole life, and designing programs and coding is similar. We notice the audience, figure out how to present the information and design the task, and focus on the user. Building software is writing; refactoring code is editing.”

Continuous learning is another part of the job Samantha loves. “Every day is a different challenge,” she says. “I get to work with new technologies, learning more every step of the way. I am part of a team with great people, and I always feel valued. I started around Christmas a few years ago, and they invited me to their holiday party before I even started. Any idea I have is considered by my team. That’s been true from the first day I walked in the door.”

The best advice Samantha has received was from one of her software engineering instructors, she says: “If you are the smartest person in the room, you’re in the wrong room. Keep learning. Stay open — even if you’re scared or not feeling confident.”

Samantha’s advice to people considering a career shift is, “I wished I had jumped into coding earlier, when I was first drawn to it. Making a change doesn’t have to be a scary obstacle. Take it in steps, and know that everyone is learning all the time.”

How to select, gather, clean, and test data for machine learning systems

How Data Becomes Insight:

The Right Data Matters

What goes into selecting, gathering, cleaning and testing data for machine-learning systems?

Q: How do you go from lots of information to usable data in a machine-learning system?

Q: This sounds difficult.

Q: If you are working with HR and payroll data, doesn’t it have a lot of personal information about people? How do you handle privacy and confidentiality issues?

Q: Once you have your initial data set, how do you know when you need or want more?

Learn more by getting our guide, “Proving the Power of People Data.”

Performance implications of misunderstanding Node.js promises

Promise.allpocalypse

The performance implications of misunderstanding Node.js promises

Benchmarking unbounded promise scenarios

for…of and for await…of

Promise.all()

Limiting simultaneous promises

Conclusion

We All Want to Belong at Work

We All Want to Belong at Work

Are you in the right job? Finding your passion and advancing your career.

Tech & Innovation Blog

Stay in Touch

On Elephant Graveyards: How bright engineers can recognize and avoid the traps

Tech & Innovation Blog

Defining career growth

Conditions required for an elephant graveyard to form

What to do if you find yourself in an elephant graveyard

Stay in Touch

Designing a Happy Career and Loving Your Work #GHC19

Tech & Innovation Blog

Stay in Touch

From Horses and Manatees to Coding

How Data Becomes Insight:

The Right Data Matters

What goes into selecting, gathering, cleaning and testing data for machine-learning systems?

Q: How do you go from lots of information to usable data in a machine-learning system?

Q: This sounds difficult.

Q: If you are working with HR and payroll data, doesn’t it have a lot of personal information about people? How do you handle privacy and confidentiality issues?

Q: Once you have your initial data set, how do you know when you need or want more?

Learn more by getting our guide, “Proving the Power of People Data.”

Promise.allpocalypse

The performance implications of misunderstanding Node.js promises

Benchmarking unbounded promise scenarios

for…of and for await…of

Promise.all()

Limiting simultaneous promises

Conclusion

We All Want to Belong at Work

Tech & Innovation Blog

Share this post

Stay in Touch

Tech & Innovation Blog

Defining career growth

Conditions required for an elephant graveyard to form

What to do if you find yourself in an elephant graveyard

Share this post

Stay in Touch

Tech & Innovation Blog

Share this post

Stay in Touch