Running things in parallel using HTML service was a brief intro on how to run a number of things at once, orchestrating executing using Google Apps Script HTML service. You may have already tried Parallel implementation and getting started or even Running things in parallel that need oAUTH2 – datastore. Now here’s something a little more complicated.

In CryptoJS libraries for Google Apps Script I mentioned that they were kind of slow under Apps Script – depending on the cryptography – up to a second for each one to encrypt/decrypt. Testing this kind of thing is pretty tricky when you run out of quota all the time. Here’s how I tested thousands at once with no quota problems.

The process

I was able to orchestrate the parallel running of all this using Parallel process orchestration with HtmlService
Here’s a snap of the run – We got 40 minutes of uninterrupted processing over 13 minutes, and processed 16,000 encrypt/decrypt pairs. To speed it up even more all I need to do is to create additional processing threads in the profile.

Orchestration profile

The action is controlled by an orchestration profile.  But there are some wrinkles in this one, because I wanted to make it dynamic, depending on the size of the run. Here’s how it’s built up. Creating the orchestration is a good first step. We don’t even have to write the functions that do the work at this stage.

Create the test data

I’m creating 4000 test items. 2 threads will do for this trivial process.

Reduce the results

After a parallel map operation, you always need to reduce the results. This ensures you have removed any dependencies between how the previous step was executed. I use a standard function for all reduce functions.

Do the encryption

I’m applying 4 different kind of encryption to each of the randomly generated messages and passwords that were created in the first process. I’m going to give 2 threads to each encryption method, so there will be 8 threads, each of which will handle 2000 encryptions. I’ll explain later what we’re going to use the options for.

Reduce the results

Again we need to reduce the results. We can just use the same reduction profile as previously

Check that it worked

Decrypting everything and checking against the original messages will show that the operation worked. This time I’ll go for 6 threads. I’ll explain the option values later.

Reduce the results

Again we need to reduce the results. We can just use the same reduction profile as previously.

Log the results

We want to make sure that we ended up with the same number of items that we started with, so do a simple summary  of the reduced data. Most of the options here are about Database abstraction with google apps script, which I always use to simplify writing data to spreadsheets.

Here’s what was logged

Put it all together

Now we’ve created a profile for each section, we can put it all together like so. I’ll repeat the whole function at the end of this page

The executor functions

These are called to do the work by the profiles we set up above. Each functionName property should point to one of these. In all cases executor functions are passed 2 parameters
  • The options you set up in your profile
  • The data from the previous stage (if there was any)

Creating the test data.

This creates a chunk of random test messages and passwords – volume as defined in options.scale.

Reduce the data

Used in multiple steps – takes the results of the previous map processes, and reduces them to a single result.

Do some encryptions

A number of these will be run in parallel threads. Since the data it will be passed will be all the test data, it needs to have a mechanism of deciding which part of the test data it will work on. In options we passed index and threads properties. From this it can select a section of the testdata unique to this this thread.
A more sophisticated algorithm might include adjusting the number of items for this thread according to the estimated complexity, but I’m just doing a simple equal distribution between threads. It will return the encrypted message, plus everything a future decrypt function will need to validate that it worked.

Decrypt and check

This will receive the reduced encrypted data, decrypt it, and check against the original message. Is uses the same allocation method as the encryption function for deciding which data to work on.v

Logging the results

I’m using the very useful crossFilter library to summarize the number of items by cipher that were checked, and outputting the result to a sheet using Database abstraction with google apps script

The whole profile

You don’t need to change any of the stuff you have already set up in Running things in parallel using HTML service, except to load this profile in Code.gs
Here’s the code for creating the profile for this all together.

and here is the profile it produces

 

or more snippets like this see Google Apps Scripts snippets