In this post, I'll dive deep into the challenges of dependency management in monorepos and share powerful solutions we discovered along the way. You'll learn about:

• Common dependency management pitfalls we encountered and how to troubleshoot them

• Real-world bugs that drove our decision to change package managers

• How to implement centralized dependency version control across all packages

• Techniques to safeguard against dependency conflicts before they occur

While our journey ultimately led us from Yarn v1 (Yarn Classic) to pnpm, the insights apply to any monorepo setup regardless of your current package manager.

Note: While these issues are specific to monorepo setups, the lessons are valuable even if you're working with a traditional repository structure.

👻 The Phantom Dependency Problem

TLDR; Phantom dependencies are the ghosts haunting your monorepo—modules your code secretly uses without declaring, causing everything to work fine today but mysteriously break tomorrow.

A Real-World Example

What seemed like a simple task—removing an unused dev dependency—turned into an unexpected nightmare that exposed a fundamental flaw in our dependency management strategy.

I was tasked with removing cypress from our monorepo because we no longer used it for E2E tests. It seemed simple—just delete some code and take cypress out of the package.json file. As a dev dependency, it was only used for tooling and wasn't bundled with our application, so I assumed it wouldn't even require QA testing. But, oh boy, was I wrong!

🛠️ The Setup: A Simple Monorepo

Let's examine a typical npm monorepo with two internal packages:

• @certa/platform: The entry point for our React application

• @certa/common: A utility package with commonly used functions

Our application simply displays the current date in a specific format:

Here is the project structure:

packages/
├── common/
│   ├── src/
│   │   └── index.js
│   └── package.json
└── platform/
    ├── src/
    │   ├── App.js
    │   └── index.js
    └── package.json
package.json

In our root package.json, we have:

{
  "name": "npm-monorepo-example",
  "scripts": {
    "start": "npm run start --workspace=@certa/platform"
  },
  "devDependencies": {
    "cypress": "^5.0.0"
  }
}

For @certa/common, we have:

// packages/common/package.json
{
  "name": "@certa/common",
  "module": "src/index.js",
  "dependencies": {}
}

And the index.js file contains:

// packages/common/src/index.js
import { format } from "date-fns";

// date-fns v1 format
export const DATE_FORMAT = "DD/MM/YYYY";

export const formatDate = (date) => {
  return format(date, DATE_FORMAT);
};

Hold on! Did you notice that date-fns isn't defined in @certa/common's package.json or the workspace root? Yet, the code still works! This is a phantom dependency—an import that shouldn't work but does. We'll explain why shortly.

For @certa/platform, we have:

// packages/platform/package.json
{
  "name": "@certa/platform",
  "scripts": {
    "start": "react-scripts start"
  },
  "dependencies": {
    "@certa/common": "^1.0.0",
    "date-fns": "^2.30.0",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "react-scripts": "^5.0.1"
  }
}

And here's App.js:

// packages/platform/src/App.js
import { formatDate } from "@certa/common";

export default function App() {
  return <h1>Print date: {formatDate(new Date())}</h1>;
}

After running npm install, here's how date-fns is structured in node_modules:

node_modules/
└── date-fns@1.30.1
packages/
├── common/
│   └── ... date-fns@1.30.1 (inherited)
└── platform/
    └── node_modules/
        └── date-fns@2.30.0

🕵️ The Mystery: Where Did date-fns@1.30.1 Come From?

If you peek into the root node_modules folder, you'll find many dependencies not explicitly listed in your workspace. This happens because npm flattens the node_modules structure, allowing access to all packages regardless of what's in your package.json.

In our case, date-fns@1.30.1 is a transitive dependency of cypress. Specifically, cypress depends on @cypress/listr-verbose-renderer, which depends on date-fns@1.30.1.

A transitive dependency is a library that your code indirectly relies on because one of your direct dependencies requires it to function properly.

🤨 The Common Misconception About Dependencies

Many developers assume that if a dependency like date-fns is added to the application entry point (@certa/platform), the same version will be used throughout the application. This is false.

In reality:

• The import { format } from "date-fns" in @certa/common uses date-fns@1.30.1

• An import in App.js of @certa/platform would use date-fns@2.30.0

This happens because Node.js resolves dependencies from the nearest node_modules folder.

🧨 The Breaking Change: Removing a Dev Dependency

Let's remove "cypress": "^5.0.0" from the root package.json and run our application:

npm start

Our application breaks! 💥 But why? 😱

After removing cypress, the dependency structure changes:

node_modules/
└── date-fns@2.30.0
packages/
├── common/
│   └── ... date-fns@2.30.0 (inherited)
└── platform/
    └── node_modules/
        └── date-fns@2.30.0

The version of date-fns in @certa/common changed from v1.30.1 to v2.30.0!

This inconsistency stems from npm and Yarn Classic's behavior of hoisting dependencies to the root. While this optimization saves disk space, it creates unpredictable dependency resolution.

The source code for this example can be found here.

🧐 Finding a Stricter Package Manager

This dependency version inconsistency was a real eye-opener, and it made us determined to prevent it from happening again. We wanted tighter control over dependencies, ensuring that only the packages explicitly listed in package.json were available in our monorepo.

Since npm and Yarn Classic were designed with different principles, they don't offer a solution to this problem. Thus began our quest for an alternative package manager.

We compared the two main alternatives: Yarn Berry (v2+) and pnpm. Both have reached feature parity, so our decision came down to:

• How the node_modules folder ensures dependency strictness

• Installation speed

• Storage consumption

• Migration difficulty

There are already many comprehensive blog posts like this and this comparing the two in detail, so I’ll not dive into the details.

We chose pnpm for these reasons:

• Strict dependency access by default - exactly what we needed

• Superior performance - we measured a 70% installation speed boost over Yarn Classic check the benchmarks in our monorepo

• Compatibility with existing tooling - unlike Yarn Berry's radical approach of eliminating node_modules entirely, pnpm works with existing Node.js tools out-of-the-box

🔒 How pnpm Ensures Strict Dependency Access

pnpm took inspiration from npm v2's nested structure but implemented it using symbolic links:

node_modules
└── .pnpm
    ├── bar@1.0.0
    │   └── node_modules
    │       └── bar -> <store>/bar
    │           ├── index.js
    │           └── package.json
    └── foo@1.0.0
        └── node_modules
            └── foo -> <store>/foo
                ├── index.js
                └── package.json

The node_modules/.pnpm directory itself (where dependencies are actually stored) is a flat structure.

Only packages explicitly listed in your package.json (just cypress in our example below) are visible in the top-level node_modules folder. Indirect dependencies remain hidden:

The .pnpm directory contains all installed dependencies, including multiple versions:

This structure guarantees strict access to only referenced dependencies. Learn more about it here.

🎏 The Dangers of Multiple Dependency Versions

Imagine you need to upgrade a library to a newer version.

When upgrading libraries in a large monorepo (which often contains 10+ packages), teams frequently opt to update one package at a time. This incremental approach is more manageable but can lead to using multiple versions of the same dependency in a single application.

So, can we use multiple versions of a dependency in a single bundled application?

Simple answer is: Yes you can, but it comes at a cost. The cost is:

1. Increased bundle size: The multiple versions will have to be bundled in your production build

2. Risk of incompatibility: Different versions of a dependency may have breaking API changes and could lead to unexpected behavior or errors in your application

The increased bundle size is obvious, but incompatibility issue is not, so let’s explore two specific issues we encountered.

🧩 Incompatible APIs Between Versions

It's not always clear when different versions of date-fns are being used across various internal packages, especially when these packages are managed by different teams. Library APIs often change from one version to another, which can lead to problems.

Let's dive into an example to illustrate how these API changes can cause issues. We'll modify the App.js example above to show this in action.

In @certa/common, we have defined DATE_FORMAT constant to store the data format string. We'll use this constant in the format function imported from @certa/platform to display the date in a different locale.

// packages/common/src/index.js

// data-fns v1
import { format } from "date-fns";

// date-fns v1 format
export const DATE_FORMAT = "DD/MM/YYYY";

export const formatDate = (date) => {
  return format(date, DATE_FORMAT);
};

// packages/platform/src/App.js
import { formatDate, DATE_FORMAT } from "@certa/common";

// data-fns v2
import { format } from "date-fns";
import { de } from "date-fns/locale";

export default function App() {
  return (
    <div>
      <h1>Print date: {formatDate(new Date())}</h1>
      <h1>Print date: {format(new Date(), DATE_FORMAT, { locale: de })}</h1>
    </div>
  );
}

Let’s run the app.

Looks like there’s an error. ❌ Turns out the format syntax changed from date-fns v1 to v2 . The format() API expects v2 format (dd/MM/yyyy) but was given v1 format (DD/MM/YYYY).

It's not immediately obvious when different versions of date-fns are used across various internal packages, especially if those packages are managed by different teams.

Errors like these are quite likely to happen. At Certa, we've learned valuable lessons from such experiences. Now, we make sure not to use multiple versions of a dependency unless it's absolutely necessary.

⛓️ Multiple Dependency Instances Problem

Another serious issue occurs with libraries that use React Context. Here's a real example we faced when upgrading react-intl from v5 to v7:

In our application, we use react-intl for internationalization across our platform. We planned to upgrade from react-intl@5.8.0 to react-intl@7.1.1.

Imagine a monorepo with just two packages. What happens if you use different versions of a dependency in each package? This might occur if you're trying to upgrade gradually, focusing on one package at a time. Let's explore the impact of this approach.

After updating only @certa/common to react-intl@7.1.1, here's how the code for both packages would look.

@certa/common

// packages/common/package.json
{
  "name": "@certa/common",
  "dependencies": {
    "react-intl": "^7.1.1"
  }
}

// packages/common/src/Welcome.jsx
import { useIntl } from "react-intl";

export const Welcome = () => {
  const intl = useIntl();

  return (
    <h1>
      {intl.formatMessage({
        id: "myMessage",
        defaultMessage: "Hello world"
      })}
    </h1>
  );
};

@certa/platform

// packages/platform/package.json
{
  "name": "@certa/platform",
  "scripts": {
    "start": "vite",
    "build": "vite build",
    "preview": "vite preview"
  },
  "dependencies": {
    "@certa/common": "workspace:*",
    "date-fns": "^2.30.0",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "react-intl": "^5.8.0"
  },
  "devDependencies": {
    "vite": "catalog:",
    "@vitejs/plugin-react": "^4.3.4",
  }
}

// packages/platform/src/App.js
import { IntlProvider } from "react-intl";
import { Welcome } from "@certa/common/src/Welcome.jsx";

// Translated messages in French with matching IDs to what you declared
const messages = {
  myMessage: "Hello world, welcome to the Certa platform!"
};

export default function App() {
  return (
    <IntlProvider messages={messages} locale="fr" defaultLocale="en">
      <Welcome />
    </IntlProvider>
  );
}

We’re using the <IntlProvider> in @certa/platform and importing a component from @certa/common that consumes that context.

After running pnpm start, we see that the app is running fine.

Nice! 😊 Now we run our CI checks and deploy it.

[React Intl] Could not find required intl object. needs to exist in the component ancestry.

Users immediately start reporting errors in our application. ❌ 😱

Why did it work in development but fail in production? In our case, we were using Vite, which handles dependencies differently between development and production. In development mode, Vite uses a looser module resolution strategy that sometimes masks dependency conflicts. However, in production, the optimization process creates a more strictly isolated bundle that exposes these incompatibilities.

The main issue: pnpm installs both versions of react-intl:

node_modules/
└── .pnpm/
    ├── react-intl@5.25.1_react@18.3.1_typescript@4.9.5
    └── react-intl@7.1.1_react@18.3.1
packages/
├── platform/
│   └── react-intl@^5.8.0 (symlink -> react-intl@5.25.1_react@18.3.1_typescript@4.9.5)
└── common/
    └── react-intl@^7.1.1 (symlink -> react-intl@7.1.1_react@18.3.1)

Notice that there are two separate copies of react-intl installed. This means the <IntlProvider> context that useIntl() in @certa/common tries to access is different from the one wrapping our application in @certa/platform. A workaround could be to re-export the useIntl hook from @certa/platform and use it across other packages in the monorepo.

This isn't something a developer would easily spot, so it's wise to avoid using multiple versions of the same dependency whenever possible.

The source code for this example is here.

🏠 Centrally Managing Dependency Versions

pnpm offers a powerful feature called “Catalogs“ that allows you to define dependency versions centrally in the pnpm-workspace.yaml file. This ensures consistency across all packages in your monorepo.

Setting it up is straightforward—just run pnpx codemod pnpm/catalog to automate the process.

Dependency versions are defined in the pnpm-workspace.yaml file at the root of the workspace.

# pnpm-workspace.yaml
packages:
  - packages/*

catalog:
  # Can be referenced through "catalog"
  cypress: ^5.0.0
  date-fns: ^2.30.0
  react: ^18.2.0
  react-dom: ^18.2.0
  react-scripts: ^5.0.1

catalogs:
  # Can be referenced through "catalog:old"
  old:
    date-fns: ^1.30.1

Now in your package.json files, you can reference these centralized versions.

When adding a dependency, we can use the catalog: protocol instead of specifying the version range directly.

{
  "name": "@certa/platform",
  "dependencies": {
    "@certa/common": "workspace:*",
    "date-fns": "catalog:",
    "react": "catalog:",
    "react-dom": "catalog:",
    "react-scripts": "catalog:"
  }
}

{
  "name": "@certa/common",
  "dependencies": {
    "date-fns": "catalog:old"
  }
}

📖 Benefits of Catalogs

• Faster dependency additions: Simply reference the catalog: protocol for existing dependencies

• Consistent versioning: Eliminate accidental version mismatches from typos

• Easier upgrades: Update one line in pnpm-workspace.yaml instead of modifying multiple package.json files

The source code for this example is here.

🚓 Enforcement

Centrally defining dependency versions is great, but how can you ensure that specific versions aren't set in individual packages? Enter syncpack, a handy tool to keep everything in check!

pnpm add -wD syncpack

Add the following script command to the root package.json file:

{
  "scripts": {
    "syncpack": "syncpack list-mismatches"
  }
}

Here is how the .syncpackrc config file will look:

{
  "versionGroups": [
    {
      "label": "Use workspace protocol when developing local packages",
      "dependencies": [
        "$LOCAL"
      ],
      "dependencyTypes": [
        "prod",
        "dev"
      ],
      "pinVersion": "workspace:*"
    },
    {
      "label": "Use catalog:old protocol for all dependencies",
      "packages": [
        "@certa/common"
      ],
      "dependencies": [
        "react-intl"
      ],
      "dependencyTypes": [
        "prod",
        "dev"
      ],
      "pinVersion": "catalog:old"
    },
    {
      "label": "Use catalog protocol for all dependencies",
      "dependencies": [
        "**"
      ],
      "dependencyTypes": [
        "prod",
        "dev"
      ],
      "pinVersion": "catalog:"
    }
  ]
}

This setup ensures that:

• Internal packages use workspace:*

• Specific packages use catalog:old for designated dependencies

• Everything else uses catalog:

This will ensure that your monorepo is following a single version policy. You can customize the configuration to allow different versions of a dependency if needed.

Running pnpm syncpack will flag any violations:

Fixing the issue by moving the version to pnpm-workspace.yaml:

Run this check in your CI pipeline to maintain dependency consistency across your project.

Conclusion

Dependency management issues like phantom dependencies and version conflicts are common headaches in monorepo setups. By understanding these problems and implementing the solutions outlined in this post, you can take the lead in your organization and become the hero 🦸 who saves your application from dependency hell.

The key takeaways:

• Be aware of phantom dependencies and their risks

• Consider pnpm for strict dependency management

• Avoid multiple versions of the same dependency when possible

• Use centralized version management with Catalogs

• Enforce dependency standards with tools like syncpack

If you found this post helpful, please share it with your network! 💻

We're hiring!

If solving challenging problems at scale in a fully-remote team interests you, check out our careers page and apply for the position that matches your skills and interests!

Many blogs already cover classic React application optimisation techniques, like image lazy loading, route level code splitting, virtualisation, useMemo, and more. This blog digs deeper and explores lesser-known techniques that are often overlooked, but can help improve your React application’s performance. Even if you don’t use React, most of these tips are still useful.

Route-level code splitting is a well-known optimisation to significantly improve load times, and we already had it in our application. We needed something extra to reduce load times even further.

🚀 Check out the improvements we got:

Certa’s platform frontend is a client-side application built using React + TypeScript + Vite. It’s a fairly large application with various third party libraries installed. If you’re using a similar stack the following tips should be very straightforward to implement.

Let’s dig right in!

🧹 Remove Old Unused Routes

Startups like ours are supposed to iterate quickly. We often leave behind a lot of unused legacy code because we're focused on building the next big thing.

As an application evolves, we tend to forget about old, unused routes that remain in our routing declarations. These old routes add to your application's bundle size over time, so it's important to clean them up once customers have fully migrated to the new version.

We're using React Router, so simply removing these routes from wherever you've declared all your routes will eliminate them from the bundle due to tree shaking.

<Route path="/" element={<MainLayout />}>
    <Route index element={<Home />} />
    <Route path="about" element={<About />} />
    <Route path="login" element={<Login />} />
    <Route path="insights" element={<Dashboard />} />
    <Route path="messages" element={<Messages />} />

    {/* Old Routes */}
-   <Route path="dashboard" element={<DashboardOld />} />
-   <Route path="chat" element={<Chat />} />

    {/* 404 Route */}
    <Route path="*" element={<NotFound />} />
</Route>

Clean up the tech debt

Go ahead and clean the code associated with the removed route. I’d recommend using a tool like knip which can find unused files, dependencies, and even exports. Best part is that it can also clean up that code automatically for you.

A cleanup is great for engineering productivity.

• A cleaner, more organised codebase reduces cognitive load for developers, making it easier to onboard new team members and allowing existing developers to work more efficiently.

• Less code to refactoring in cases of large repo wide refactors.

🛒 Lazy Load Large Third-Party Libraries

When building a large application, it’s inevitable to use many open source libraries. Some of these libraries tend to be quite large. It makes more sense to lazy load these libraries so that they are fetched only when a component importing them is rendered.

Let’s take the example of jsoneditor-react. We have included it in our application for JSON editing capabilities but these capabilities are only required on certain pages and so it doesn’t make much sense to load them initially on page load. jsoneditor-react contributes around 210kB (gzipped) to the bundle size which is significant enough to lazy load.

Following example of lazy loading jsoneditor-react is a rather more extreme case where multiple dependent libraries also have to be loaded in order for the library to work.

import loadable from "@loadable/component";
import type {
  JsonEditor,
  JsonEditorProps as JsonEditorComponentProps
} from "jsoneditor-react";
import Ajv from "ajv";

const ajv = new Ajv({ allErrors: true, verbose: true });

const Editor = loadable(
  () =>
    import("ace-builds/src-noconflict/ace")
      .then(() =>
        Promise.all([
          import("jsoneditor-react"),
          import("brace"),
          import("jsoneditor-react/es/editor.min.css"),
          import("brace/mode/json")
        ])
      )
      .then(([{ JsonEditor }, { default: ace }]) => {
        return forwardRef<JsonEditor, JsonEditorComponentProps>(
          (props, ref) => (
            <JsonEditor ace={ace} ajv={ajv} ref={ref} {...props} />
          )
        );
      }),
  {
    fallback: <FallbackLoader />
  }
);

// Usage
const MyForm = () => {
    return (
        <Editor ... />
    )
}

We’re using the loadable-components library to make lazy loading simpler. It essentially only fetches the module when the component mounts (<Editor> is this case).

Here is a simpler use-case where we need the xlsx library capabilities to export an excel sheet.

export const exportExcel = async (
  data: Record<any, any>[],
  fileName: string,
  extension: string
) => {
  // Dynamic import
  const XLSX = await import("xlsx");

  const worksheet = XLSX.utils.json_to_sheet(data);
  const workbook = XLSX.utils.book_new();
  XLSX.utils.book_append_sheet(workbook, worksheet, "Sheet1");
  XLSX.writeFile(workbook, `${fileName}.${extension}`, {
    compression: true
  });
};

// Usage
const MyForm = () => {
    return (
        <button onClick={() => exportExcel(...)}>
          Export as excel sheet
        </button>
    )
}

🌲 Use Tree Shakable Alternatives

The lodash library is not tree-shakable by default. You need to import modules with specific paths like lodash/map so that you don’t end up importing the whole library. With a large team, it’s not always possible to follow specific import syntax patterns. One mistake can lead to an increased bundle size.

The lodash-es package on the other hand is tree-shakable by default as it uses ES modules. So you can import without accidentally including unnecessary code in the bundle.

// All lodash modules included in bundle
import { debounce } from 'lodash';

// Only 'debounce' function included in bundle
import debounce from 'lodash/map';

// Only 'debounce' function included in bundle
import { debounce } from 'lodash-es';

// Only 'debounce' function included in bundle
import debounce from 'lodash-es/debounce';

If your team can’t move away from lodash, you can always use an ESLint rule:

"no-restricted-imports": [
   "error",
   {
      "paths": [
         {
            "name": "lodash",
            "message": "Import [module] from lodash/[module] instead"
         }
      ]
   }
],

🔌 Pre-connect to Frequently Used Endpoints

In your main HTML file, you can define origins using <link rel="preconnect"> which will tell the browser to preemptively perform all of the handshake (DNS+TCP for HTTP, and DNS+TCP+TLS for HTTPS origins) for those origins. This will speeds up future loads from a given origin.

These origins should be cross-origin. It has no benefit on same-origin requests because the connection is already open.

These origins are usually ones that you need on page load and are used on almost all pages of your application.

Here is an example:

<!-- Preconnect - Full connection including DNS, TCP, and TLS -->
<link rel="preconnect" href="https://www.gstatic.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://www.recaptcha.net" crossorigin />
<link rel="preconnect" href="https://api.certa.in" />

The crossorigin attribute is added when the server expects CORS headers in the request. If the server is cross-origin but doesn't require CORS headers (like for images or scripts that don't need CORS), you should omit the crossorigin attribute else the pre-connection would be of no use and the browser would be forced to establish a connection again.

Before preconnect:

After preconnect:

That’s around 0.4s faster page loads already. 🚀

🖌 Optimize Fonts

Eliminate Extra Fonts (if possible)

Our application used 4 different fonts before optimization. One was our primary font, others were just used at single places in the UI. We redesigned our UI in a way that eliminated the need for some most of the extra fonts. Loading fonts is blocking in nature, so lesser the fonts, the better for page loads.

Preload Fonts

When importing fonts in HTML, we typically import like this:

<link
  rel="stylesheet"
  href="https://fonts.googleapis.com/css2?family=Inter:wght@100..900&display=swap"
/>

Adding a preload tag tells the browser to start downloading the asset early and treat it as high priority. It ensures that the font is available to be used before browser renders the page. This prevents UI flicker caused by loading default browser fonts and then the desired fonts.

<link
  rel="preload"
  as="style"
  href="https://fonts.googleapis.com/css2?family=Inter:wght@100..900&display=swap"
  crossorigin
/>
<link
  rel="stylesheet"
  href="https://fonts.googleapis.com/css2?family=Inter:wght@100..900&display=swap"
/>

🪑 You might not need those Polyfills

A polyfill is a piece of code that provides modern functionality to older browsers that don't support it natively. For example, if you want to use Array.prototype.includes() (ES7 feature) in a very old browser that doesn't support it, you'd need a polyfill to replicate that functionality.

Polyfills do come with several downsides:

• Bundle Size: Each polyfill adds to your JavaScript bundle size, increasing download times and potentially affecting performance.

• Runtime Overhead: Polyfills often can't match the performance of native implementations.

• Maintenance Burden: Managing polyfills adds complexity to your build process and dependencies.

Before including polyfills, ask yourself:

• What browsers do your users use?

• What percentage of your users would benefit from the polyfill?

• What's the performance impact of including the polyfill?

It might be beneficial to drop support for old browsers and ask them to upgrade if the users using old browsers is a very small fraction.

At Certa, we were using polyfills for string.prototype.matchall and had react-app-polyfill package added to polyfill ES6 features for Edge 17 and 16 compatibility.

After analysing the browsers our users were using, we found that those browser are no longer used and proceeded to clean them up.

🪆 Manual Chunking with Vite

While Vite provides automatic chunking out of the box for source code, there are scenarios where manual chunking gives you more control over your build output. Allow me to explain:

By default Vite third-party libraries and source code is bundled together in what we call the index chunk. This chunk is the first chunk to load and is always loaded no matter the route.

We should aim to break down large chunks in order to reduce fetching delays.

• Aim for chunks between 100-300KB

• Too many small chunks can increase HTTP requests

• Too few large chunks can delay initial load

Visualize your Bundle

Before we optimize chunking, it would be great to visualize it first so that we can observe the difference. Using rollup-plugin-visualizer, we can view a neat graph of all the chunks and how much size they take up.

Separate Chunk for Third-Party Libraries

Third-party packages are not dependent on any application source code, so they can all be easily separated into a separate chunk without any breakages. Let’s call this chunk the vendor chunk.

You can further divide the vendor chunk into multiple separate chunks. Be careful here, because third-party libraries are often inter dependent on each other, so it’s important to keep inter dependent packages in the same chunk else they might fail to load at run time.

Possible scenario that will lead to an error:

// Will FAIL

// Dependency graph
@lexical/core -> @lexical/react -> @lexical/core

// Manual chunking
if (id.includes("@lexical/react")) return "lexical-react";
if (id.includes("@lexical/core")) return "lexical-core";

// Output chunks
lexical-react-[hash].js
lexical-core-[hash].js

In above example, we are keeping to inter-dependent packages in separate chunks.

When lexical-react chunk is loaded and parsed, it will see that there is an import to the lexical-core chunk. Parsing will be paused for "lexical-react" and move to loading and parsing “lexical-core“ chunk.

The lexical-core chunk will have an import statement defined for lexical-react. But the lexical-react chunk is still not loaded yet, so we’re stuck in a deadlock and the parsing fails and error is thrown.

For this reason, we need to keep related packages in the same chunk.

// Will SUCCEED

// Dependency graph
@lexical/core -> @lexical/react -> @lexical/core

// Manual chunking (will include both `@lexical/core` and `@lexical/react` in same chunk)
if (id.includes("lexical")) return "lexical";

// Output chunks
lexical-[hash].js

It’s not recommended to implement manual chunking in source code due to high likelihood of creating circular dependencies that will break your app too often. The right way to create separate chunks in source code is to use dynamic imports where is makes sense. Each route should ideally be dynamically imported so that the module bundler knows to analyse the dependency graph and separate the chunks accordingly.

Here is the full manual configuration:

// vite.config.ts
const viteConfig = defineConfig(() => ({
  build: {
    rollupOptions: {
      output: {
        manualChunks
      }
    },
    ...
  },
  ...
}))

function manualChunks(id: string): string | void {
  const isVendor = id.includes("node_modules");
  if (isVendor) {
    // Creates manual chunks from node_module deps. Reducing the vendor chunk size.
    if (id.includes("core-js")) return "core-js";
    if (id.includes("react-flow-renderer")) return "react-flow-renderer";
    if (id.includes("recharts")) return "recharts";
    if (id.includes("lexical")) return "lexical";
    if (id.includes("d3-")) return "d3";
    if (id.includes("lodash-es")) return "lodash";
    if (id.includes("moment")) return "moment";
    if (id.includes("dexie")) return "dexie";
    if (id.includes("date-fns")) return "date-fns";
    if (id.includes("react-popper")) return "react-popper";
    if (id.includes("handlebars")) return "handlebars";
    if (id.includes("@sentry")) return "sentry";
    if (id.includes("@radix-ui")) return "radix-ui";
    if (id.includes("@dnd-kit")) return "dnd-kit";
    if (id.includes("jsoneditor-react") || id.includes("brace"))
      return "jsoneditor";

    // Default chunk name for third-party packages
    return "vendor";
  }
}

All chunks required for initial page load are loaded in parallel. So the largest chunk can be the bottleneck, which is where manual chunking helps to break down chunk further.

Conclusion

Most of the above techniques are framework independent. They can be done one at a time and are fairly straightforward to implement. Enhancing the performance of your application will greatly improve user experience, leading to higher satisfaction and engagement.

With a combination of all these techniques, we benchmarked major improvements in metrics across our platform and saw a 36% improvement in page load times on average. Let’s dig into the metrics, shall we?

• 57% reduction in the app size loaded on Login page

• LCP (Largest Contentful Paint): 27.2% faster

• FCP (First Contentful Paint): 28.5% faster

• FMP (First Meaningful Paint): 30% faster

• DOMContentLoaded (DCL): 41.1% faster

Hope you achieve similar or even better results! 🚀

We're hiring!

If solving challenging problems at scale in a fully-remote team interests you, head to our careers page and apply for the position of your liking!

The First Signs of Trouble 📞

On Tuesday, we noticed a minor uptick in database activity. The Read IOPS on the main database started climbing. Initially dismissed as a minor fluctuation, after monitoring it for some hours, it was clear something was wrong. The database was straining under an unexpected load, and the effects soon rippled through the system.

The Impact Spreads 🧨

As the week went on, our DevOps team began receiving alerts about high-read IOPS. They sprung into action involving a few platform engineers, and promptly began the investigation. Meanwhile, several clients reported high latency across multiple features to our customer success team.

The Investigation Begins 🕵🏽‍♂️

To identify the root cause, we delved into the monitoring dashboards, examining metrics and logs to understand what was driving the increased read IOPS. When we compared metrics from the last two weeks, we saw a steep rise in the read IOPS count, with no sign of decline.

The Discovery 💡

Further investigation revealed a troubling execution plan within one of the database schemas. An EXPLAIN ANALYZE query showed some operations were taking far longer than expected, leading to a cascade of delays and increased read IOPS. In such cases, VACUUM ANALYZE always comes as a lifesaver, so we ran the command to refresh the stats and generate an optimized query plan. When we checked the dashboards for clues, we noticed that the autovacuum command was running continuously. We focused on it and checked what was happening. The database automatically triggers this command whenever the bloat size crosses a threshold limit, and when we checked the stats, we found out that this was indeed what had happened with our database.

SQL query:

SELECT schemaname, relname, n_live_tup, n_dead_tup, trunc(100*n_dead_tup/(n_live_tup+1))::float "ratio%",
to_char(last_autovacuum, 'YYYY-MM-DD HH24:MI:SS') as autovacuum_date,
to_char(last_autoanalyze, 'YYYY-MM-DD HH24:MI:SS') as autoanalyze_date
FROM pg_stat_all_tables
ORDER BY n_dead_tup DESC;

More technical details for those who are interested…

Let’s take a pause and understand a few technical details before continuing the story.

• Understanding dead tuples in PostgreSQL

  When deleting rows in PostgreSQL, the database does not actually remove them but marks them as dead tuples. One might think that it would be easier to delete the row when performing the delete operation, instead of relying on something like vacuum, right? But there are reasons for not doing it, which are:

  1. Recovery and rollback

     During the execution of a DELETE operation, the database cannot just delete the record without committing the transaction because it may fail or be aborted at any point in time, leading to the need for a rollback and restoration of a previous version of the record.

  2. Concurrency Control

     • PostgreSQL uses MVCC to manage concurrent transactions without locking the database tables. Instead of locking rows, it maintains multiple versions of a tuple. This allows readers to access the previous versions of the data while writers are modifying it, enabling better performance and less contention.

     • By marking tuples as dead rather than deleting them immediately, PostgreSQL ensures that transactions running in parallel can continue to see a consistent view of the data according to their isolation level. Deleting tuples outright could violate this consistency.

     • For example, consider two transactions running in parallel.

       • Transaction T1 performs a SELECT operation on a dataset and then Transaction T2 performs DELETE on a few rows in the same dataset just after some time.

       • The view of transaction T1 is locked and does not change until it completes hence deleting those rows by another transaction would not be a good idea.

       • Instead, transaction T2 actively marks those rows as dead, allowing T1 to still see and return the previous copy so that the user can view a version of the rows at the time it was requested.

       • Now instead of returning, T1 performs any UPDATE operation after T2 marks it as dead, commits, and raises a serialization error indicating the record has been updated, forcing T1 to refresh the data before performing any update.

  3. Index Maintenance

     Marking tuples as dead simplifies the handling of indexes. Deleting tuples immediately would require updating or removing corresponding index entries right away, which could lead to additional complexity and performance overhead.

• What is “bloat”?

  Database bloat refers to the excessive accumulation of "dead tuples" or obsolete data in a database. Bloat happens when dead tuples pile up faster than the database can clean them up. For example, if the application updates or deletes 1,000 records per second, but the autovacuum daemon can only remove 800 dead tuples per second, 200 dead tuples are left behind every second, causing bloat. This is just a hypothetical example—it's not possible to fine-tune the autovacuum daemon to such specific rates.

• Problems caused by database bloat

  In Postgres, the query planner decides the best way to execute a query. It looks at table statistics, indexes, and data types to recommend the most efficient execution path. When a table is bloated, ANALYZE in Postgres can provide inaccurate information to the query planner, causing slow queries and increased disk I/O operations due to the processing of unnecessary data.

  For instance, if a table has 350 million dead tuples but only 50 million active rows, the bloat score is 7 (a 7:1 ratio of dead to active rows). This high bloat score leads the query planner to make poor decisions, resulting in slow queries. During query execution, PostgreSQL loads both live and dead tuples into memory, which increases disk I/O and degrades performance because it has to process unnecessary data.

• VACUUM and autovaccum for cleaning bloat

  VACUUM command can be used to manually clean up these dead tuples and reclaim the space. PostgreSQL also provides autovacuum functionality which automates vacuum activity. By default, Postgres triggers cleanup when dead tuples make up 20% of the table. Based on your database workload and performance requirements, you have the option to adjust various settings. However, the autovacuum process uses resources like CPU, memory, and disk I/O, which can affect the database’s regular operations.

  To learn more about this, refer to "Understanding Autovacuum in Amazon RDS for PostgreSQL Environments.”

The Search for a Solution 🔍

Now there were two questions that we were asking ourselves.

1. Why does bloat size increase so much even though autovacuum is running continuously?

2. How could we speed up the removal of table bloat from the database?

While we were still seeking answers to the first question, we didn’t want the customer experience to suffer. Therefore, we prioritized addressing the symptoms and began exploring potential solutions. Each option had advantages and disadvantages, demanding us to carefully evaluate the potential downtime and challenges, particularly in a production environment. Since finalizing a solution was going to take some time, we opted for a short-term fix by increasing provisioned IOPS to ensure a seamless customer experience.

Then, another problem was reported at the same time 😵

Everything was working fine for our clients, and we were still searching for answers to our first question while preparing a solution. Then another issue arose, requiring our immediate attention. Our analytics database, which is a read-only replica of the main database, became slow, causing one of our clients to experience problems generating daily analytic reports. Our investigation revealed that our replica database was consistently running at over 98% load. We scaled up the database vertically in order to swiftly address this issue and ensure the client could continue working without experiencing slowdowns.

What the… So, this is what happened… 🤔

After scaling our replica database, it restarted, causing a drop in CPU load. To our surprise, the bloat size in our main database also decreased rapidly. Initially, we thought this might be because of the increased IOPS capacity. However, further investigation revealed the true cause.

We discovered a few queries had been running for a long time on our replica database. Upon checking the configuration, we noticed that hot_standby_feedback was enabled, which caused the problem. Enabling the hot_standby_feedback parameter on the replica instance usually prevents vacuum from removing dead row versions needed by running queries on the replica. This prevents query conflicts or cancellations and delays the cleanup of dead rows on the primary database, which can lead to table bloat. In our case, when the read replica database restarted during the instance scaling process, It canceled all active sessions. This allowed the autovacuum process to complete in the primary instance, reducing read IOPS.

While we worked on a long-term solution, the problem recurred in a few days. This time, we checked the replica database for long-running queries and discovered the same issue. We terminated those long-running queries rather than restarting the database. Within a few hours, the bloat size returned to normal, and the application performance improved.

More technical details 🖥️

Dead tuples are created to ensure concurrency when two transactions are running in parallel and working on the same dataset. But what if those transactions are running on two different servers, the main server and a replica? Rows are marked as dead, so they will still exist, and even after replication, the replica server will know which rows are marked as dead. However, a problem arises when autovacuum runs on the main server because when these dead tuples are deleted, the same will be reflected on the replica. What if there is a transaction that still uses those rows? This will cause replication conflicts.

To prevent such conflicts, PostgreSQL provides the hot_standby_feedback toggle. When enabled, the replica periodically sends information about the oldest transaction that is running. With this information, VACUUM can delay cleaning up certain rows, thus avoiding replication conflicts. However, if a query runs for a long duration or never terminates, these dead tuples will accumulate, resulting in database bloat.

Lessons 📝

• The database is the backbone of your application, so it is crucial to configure it properly to ensure optimal performance and reliability. In our case, we had hot_standby_feedback enabled but had improperly configured the timeout parameter to terminate long-running queries. Although we had alerts enabled, these queries went unnoticed because of the incorrectly configured time parameter. We had to reconsider various factors to determine the optimal time parameter and reconfigure it accordingly.

  These parameter values may vary depending on the use case, so it is always recommended to understand your specific use case before configuring them.

• We use replica db to fetch analytic reports, while most queries we run there are reviewed first. Those long queries were the ones that were not audited and those ended up biting us in back. So we learned to always allow audited queries that won't cause resource starvation for other queries.

• Customer experience is of utmost priority. While working on long-term solutions, ensure you provide quick resolutions that make the customer’s life easier.

Additional Information 📃

We also explored the following options to clean up the bloat, but fortunately didn’t need to use:

• VACUUM FULL

  • Pros: It can be faster than VACUUM ANALYZE and will remove all bloat in the table while shrinking the table’s physical size on disk.

  • Cons: It requires an exclusive read/write lock on the table for the operation’s duration, which can cause application outages depending on the table size.

• pg_repack (reference)

  • Pros: Very fast and doesn’t require a read/write lock on the table.

  • Cons: It is very resource-intensive, which can degrade overall database performance. It can also cause significant replication lag and OutOfMemory errors when using replication slots.

• pgcompacttable

  • It performs the same function as pg_repack by modifying the rows in place.

  • However, it uses significantly fewer resources and operates much slower, making it less problematic for replication slots.

  • Despite this, pgcompacttable was too slow to effectively remove the amount of bloat we had.

References 🔗

1. Understanding Postgres IOPS: Why They Matter Even When Everything Fits in Cache

2. Analyse and Vacuum in PostgreSQL: Keep Your Database Clean!

3. Understanding the Autovacuum Process in PostgreSQL

4. Remove bloat from Amazon Aurora and RDS for PostgreSQL with pg_repack

5. The Postgres replication dilemma

6. WHAT HOT_STANDBY_FEEDBACK IN POSTGRESQL REALLY DOES

Bonus Content 🤓

In our codebase, we had a very specific parser responsible for converting meta JSON created on front-end into a format ingestible by our backend. The parser was exclusively utilized by the frontend until recently. However, while we were building our AI suite, another use-case emerged, where the backend also needed to use the same utility. Since this parser undergoes frequent updates, maintaining separate copies of the same logic in different languages is a maintenance nightmare - it's error prone & increases unnecessary dependencies between teams.

At Certa, we believe in building scalable processes - inside & out. So, we decided to build a centralized solution.

The Ideal Solution

The ideal solution to this problem would have the following characteristics:

• Seamless Integration: It had to integrate smoothly with our existing codebases, minimizing the need for extensive restructuring or managing multiple repositories.

• Maintainability: Easily maintainable and automatable, allowing for straightforward updates and integration with deployment pipelines.

• Scalability: The solution had to scale effortlessly to accommodate fluctuations in demand, ensuring consistent performance under varying loads.

• Low Latency: Ensuring low latency was paramount for optimal performance, as every millisecond counts in delivering a seamless UX. This was crucial for maintaining our high standards of customer satisfaction.

• Cost-Effectiveness: The solution needed to be cost-effective, providing a balance between performance and expense.

With these characteristics identified and a bit of research, we arrived at 3 potential solutions:

1. Shared Library

   Initially, publishing a library on NPM seemed like a straightforward solution. However, our backend is driven by Django, which presented significant hurdles. While workarounds such as wrappers or transpilers exist, they introduce unnecessary complexities. Although we could automate some of this through CI/CD pipeline, but the compatibility issues remained a concern.

2. WASM Module

   To address the language compatibility issues, we considered converting our code to AssemblyScript. We could set up a CI/CD pipeline that would allow us to compile it into a WASM module and upload it to a CDN. But this solution had a few challenges of its own. The most notable one being the latency that is introduced when passing large data from JavaScript to WASM (and vice versa). This latency would be noticeably worse when compared to function calls to the same parser, within JavaScript.

3. Serverless Functions

   Serverless functions like AWS Lambda seemed to be one of the better options, compared to the other two, for our use-case. The code can exist in our frontend repository, which can be deployed whenever it is changed and pushed. The serverless functions provided several benefits like:

   • Automatic Scaling based on demand

   • Cost-efficient operation, especially for tasks that are not constantly running

   • An option to deploy on edge for ultra low latency in all regions around the world

However, there's still the issue of cold starts. A “cold start” refers to the duration required to initialize and execute a fresh instance of a serverless function. The delay caused by cold starts, especially in scenarios with infrequent invocations, could impact user experience.

Solution: Cloudflare Workers

While exploring our serverless options, Cloudflare Workers emerged as the ideal solution for our use case, meeting all our criteria. With its 0ms cold starts, this edge service allows our backend microservices to access the transformational code almost instantaneously. Read more on how Cloudflare Workers is able to eliminate cold starts; it's inspiring to see how they've used isolates to eliminate cold-starts!

How did we use it?

Initial Setup

1. We started off by adding a new package into our front-end monorepo as this code clearly needed to be independent of the rest our front-end logic.

2. Within this package, we initialized a Cloudflare Workers project and moved the parsers' logic into it, exposing an endpoint accessible to external services.

3. To have a cleaner and maintainable code, we used itty-router library, which offered a more elegant solution compared to nested if-else statements in the Worker's fetch event handler.

4. Deployment was done using wrangler deploy command, which was installed when we initialized the Cloudflare Workers project.

   It looked something like this:

    import type { IRequest } from "itty-router";
    import { Router, json, error, createCors } from "itty-router";
   
    const router = Router({ base: "/api" });
   
    const { preflight, corsify } = createCors({
      origins: ["*"],
      methods: ["POST", "OPTIONS"]
    });
   
    router
      // embedding preflight upstream to handle all OPTIONS requests
      .all("*", preflight)
      .post("/tranform/", yourTransformController)
      .all("*", () => error(404, "Invalid endpoint"));
   
    export default {
      fetch: (req: IRequest) =>
        router
          .handle(req)
          .then(json)
          .catch(error)
          // corsify all Responses (including errors)
          .then(corsify)
    };
   

Benchmarks

After completing the initial setup and deployment of the parser on Cloudflare Workers, we conducted testing to verify its functionality. Our tests focused on assessing the round-trip time (RTT).

These tests were performed on a MacBook M1 Pro with a high-speed internet connection over WiFi.

Results

Our testing results showed that the parser was performing well within acceptable limits. Notably, the benchmarks involved a larger-than-usual payload size of 1.5MB. For typical payloads (less than 100KB), the round-trip time (RTT) was consistently around 200 ms.

Given these outcomes, we decided to proceed with this solution.

Finally, setting up CI/CD

The next step was to set up a CI/CD pipeline to ensure that any changes made to the parser would be automatically deployed on Cloudflare Workers. Deploying our project was straightforward using Cloudflare's GitHub Action, cloudflare/wrangler-action@v3.

To optimize the deployment process, we made a key modification. We configured the GitHub Action to trigger only when changes were made to the specific folder containing our parser. This targeted approach prevents unnecessary deployments and ensures that the action only runs when there are relevant updates.

on:
  push:
    branches:
      - development
    paths:
      # Only run the workflow when changes are made to our code
      - "src/parser/*"

..

Conclusion: How did we benefit from this?

• Minimal Latency: Our system consistently achieves an average round trip of around 200ms for typical use cases to transform and return data on a decent network, complemented by 0ms cold starts, ensuring consistent and fast response times for all requests.

• Reduced Maintenance: By utilizing a single codebase, we've significantly reduced maintenance efforts. Managing two repositories wouldn't just increase complexity but also add to management tasks without necessarily demanding more resources.

• Scalability: The system seamlessly handles traffic spikes, without any degradation in performance. The deployment across 200 cities ensures that we can provide consistent service to users globally.

• Cost Savings: In contrast to other serverless options, Cloudflare Workers charge based on CPU time instead of execution duration, which includes both active processing and idle waiting times. For our expected usage of 20 million requests and 100 million CPU milliseconds per month, the total expected monthly cost is only $9.40, significantly lower than alternative serverless options.

We are hiring!

If solving challenging problems at scale in a fully remote team interests you, head to our careers page and apply for the position of your liking!

As development teams rely on CI/CD tools and workflows more and more to deploy their applications, testing has become a key integral part of the SDLC. By using automated testing tools like CircleCI or GitHub Actions, we can ensure our code meets requirements, performs reliably, and is maintainable in the long run

However, as codebases increase in complexity and size with the addition of new features over time, so do the number of test cases that we write to ensure proper coverage and verification of our core business logic. With that, most development teams eventually run into the problem with flaky test cases.

In this blog, we will navigate through the various causes of flakiness within test cases and how we can aim to resolve them. For context, we will stick to python3, and Django as examples to follow along with.

How do we define flaky test cases?

Flaky test cases can be normally defined as test cases that can sometimes pass or fail without any changes to the underlying code. These test cases flake due to some form of non-determinism within the code that lead to mismatched results or undefined behavior.

Facing these flaky issues within CI pipelines is a growing problem for development teams of all sizes as it leads to hindrance in productivity, loss of trust in the CI tools, and frustrations all around. Flakiness is a characteristic of test suites that present themselves more within CI environments in general. The reason for this could be the differences in resources when comparing it to local development environments and the under-the-hood dependencies that may give way to conflicts.

Luckily, modern-day CI tools are well suited to help identify flakiness within our test suites as part of their offerings and while they may help in identifying flaky test cases, the responsibility falls upon the development teams to go in and address those issues promptly to keep the codebase clean and up to the mark.

Diagnosing the problem

Test cases can fail for a variety of reasons, as non-determinism can manifest from the most seemingly random or innocent places within the code. Having said that, when dealing with flaky test cases over a long period, it becomes easier to see the few patterns that emerge from test suites that help us identify root causes quickly and be more mindful about how we write our test cases.

Let’s dive deeper into the various causes of flaky test cases and what we can do to remedy them.

1. Asynchronous Functions

When dealing with test cases where the target functionality executes asynchronously, there may be scenarios where the expected data is not ready by the time we run our assertions, leading to flakiness. This scenario is one of the more common reasons why test cases flake.

Let’s take the example of a function that we invoke via celery.

# tasks.py
from celery import shared_task
from .models import YourModel

@shared_task
def internal_task(model_id):
    model_instance = YourModel.objects.get(pk=model_id)
    # Perform some operations on the model instance
    model_instance.some_field = 'modified'
    model_instance.save()

When writing test cases, we generally go from writing unit tests —> integration tests —> end-to-end tests. While the core functionality of our sample function is verified within unit tests, integration tests are often found to be flakier due to the asynchronous behavior we introduce within the entire flow.

In other words, if we were to invoke this function within a function-based view for a Django application, we would likely do something along the lines of

from django.shortcuts import render
from django.http import HttpResponse
from .tasks import internal_task
from .models import YourModel

def my_view(request):
    # Assume you have a model instance
    model_instance = YourModel.objects.get(pk=1)

    # Call the Celery task asynchronously
    internal_task.delay(model_instance.id)

    return HttpResponse("Task has been queued for execution.")

When writing integration tests for this view, we might encounter flakiness if the celery task does not finish execution before our assertions are made.

This is where Mocking comes in handy!

Mocking allows developers to replace parts of the code with mock objects, which simulate the behavior of real objects. This is particularly useful when testing asynchronous functions because it enables us to isolate the function being tested from its dependencies. By mocking external dependencies such as database queries or API calls, we can focus solely on testing the function's logic without worrying about the behavior of other components.

# test_views.py
from django.test import TestCase, Client
from django.urls import reverse
from .models import YourModel
from unittest.mock import patch

class MyViewTest(TestCase):
    @patch('yourapp.tasks.internal_task.delay')
    def test_my_view(self, mock_internal_task_delay):
        # Create a model instance
        model_instance = YourModel.objects.create(id=1)

        # Call the view function
        client = Client()
        response = client.get(reverse('my_view'))

        # Check if the response is successful
        self.assertEqual(response.status_code, 200)

        # Check if the Celery task is called with the correct arguments
        mock_internal_task_delay.assert_called_once_with(model_instance.id)

By doing this, we can still write our integration tests by relying on the individual unit tests to verify functionality.

2. Improper Management of State

This kind of flaky test case is more subtle as during development, it can be easily missed.

When writing new test cases, we often only test the particular suite we are working on. However, when we push our code into our CI pipelines, we run all of our test cases at once. What ends up happening is that test cases that were invoked before the new test case that was pushed as part of PR can unintentionally alter the state of the system leading to flaky test cases.

When running these test cases locally, we might think that our test case is robust but a different test case that was run before yours was responsible for altering the state of the system about which you made assumptions.

This modification of state most commonly stems from improper management of resources and states within test suites.

For example, let us take a look at the following test suite.

from django.test import TransactionTestCase
from django.utils import timezone

class TestFlakyTimeZone(TransactionTestCase):

    @classmethod
    def setUpClass(cls):
        super().setUpClass()
        # Alter global timezone without proper teardown
        timezone.activate("Asia/Kolkata")

    def test_something_with_timezone(self):
        # Test logic that relies on the altered timezone
        assert timezone.is_active is True  # Example assertion

Seemingly harmless, but on closer inspection, we find out that timezone.activate() modifies the timezone globally and is not limited to the function scope. This introduces inadvertent flakiness to any other downstream test cases that make assumptions about the timezone and hence fail due to the unfortunate ordering of the test cases.

What we can do here to mitigate such scenarios is to ensure we cater for proper cleanup of resources within our test suite. For python3, when using the in-built unittest module, we can use the tearDown() or tearDownClass() to handle the cleanup.

import pytest
from django.test import TransactionTestCase
from django.utils import timezone

class TestFlakyTimeZone(TransactionTestCase):
    @classmethod
    def setUpClass(cls):
        super().setUpClass()
        # Alter global timezone without proper teardown
        timezone.activate("Asia/Kolkata")

    def test_something_with_timezone(self):
        # Test logic that relies on the altered timezone
        assert timezone.is_active is True  # Example assertion

    @classmethod
    def tearDownClass(cls):
            timezone.deactivate()

This is just one example of where cleanup is important. However, there can be other scenarios where open files, disconnected signals, or some altered state must be reset to their original state to allow downstream tests to run smoothly.

3. Parallelism and Transactions

To expedite the execution of our test suites, it's common practice to employ various tools that facilitate the distribution of tests to run concurrently. For example, when utilizing pytest, we frequently leverage pytest-xdist to distribute tests across multiple CPUs, significantly improving execution times.

While parallel test execution offers notable performance benefits, it can introduce nuanced challenges, particularly when dealing with database transactions. This is especially evident in the case of pytest-xdist, where tests are parallelized at the function level. Let's illustrate this with an example from Django's TransactionTestCase.

TransactionTestCase encapsulates each test method within a database transaction, ensuring that the database state is reverted to its original state upon test completion. However, a common issue arises when individual test cases attempt to manipulate shared data concurrently.

The crux of the problem lies in the potential for conflicts arising from simultaneous modifications to shared data. Such conflicts can lead to test flakiness, where the outcome of a test becomes unpredictable due to interference from other concurrently executing tests.

For example -

# test_example.py
from django.test import TransactionTestCase
from myapp.models import ExampleObject

class ExampleTestCase(TransactionTestCase):
    @classmethod
    def setUpClass(cls):
        super().setUpClass()
        cls.example_object = ExampleObject()

    def test_operation_1(self):
        self.example_object.perform_operation_1()
        assert self.example_object.get_result() == expected_result_1

    def test_operation_2(self):
        self.example_object.perform_operation_2()
        assert self.example_object.get_result() == expected_result_2

Let’s visualize how the two test cases interact with the shared object :

In this example, we can observe a potential conflict that leads to flaky test cases because, in TransactionTestCase, every test method is wrapped in its transaction. However, since we defined our ExampleObject within the setUpClass method, this essentially acts as a shared resource. So, what we may run into here is an instance where while using pytest-xdist , the two test methods run in parallel but run into a database transaction error due to the shared resource being acted upon in one or the other method.

One approach to solving this would be to use the setUp() method instead.

# test_example.py
import pytest
from django.test import TransactionTestCase
from myapp.models import ExampleObject

class ExampleTestCase(TransactionTestCase):
    @classmethod
    def setUp(self):
        super().setUp()
        self.example_object = ExampleObject()

    def test_operation_1(self):
        self.example_object.perform_operation_1()
        assert self.example_object.get_result() == expected_result_1

    def test_operation_2(self):
        self.example_object.perform_operation_2()
        assert self.example_object.get_result() == expected_result_2

Let’s visualize how this change affects our test case!

It is important to note that this solution is not a silver bullet by any means, and there are a few things to keep in mind when solving concurrency-related issues for flaky test cases!

1. If we use setUp we will trade off on execution times and memory since we will instantiate the ExampleObject every time for every test method

2. There can be nuances within the code you are meaning to test that may require handling concurrency differently.

To mitigate such issues, it's essential to design test cases that isolate their data dependencies and avoid modifying shared resources whenever possible. This ensures test reliability and repeatability across different execution environments.

4. Improper Usage of Assertions

This category of flaky test cases is more of a development miss than an environmental dependency or systematic nature of execution. They can occur randomly and are a common “right-under-your-nose” error.

This is the missing semicolon equivalent for test cases after all!

When using assertions, it is important to remember the data types we are dealing with and use appropriate assert statements for varying kinds of expected values.

The most common example here is the use of assertListEqual when comparing 2 lists. Sometimes, if the order is not guaranteed for our actual_value , we can often run into flaky scenarios where our expected_value has mismatched elements.

import unittest
from my_app.utils import foo

class TestListComparison(unittest.TestCase):
    def test_assert_list_equal(self):
        expected_value = [1, 2, 3]
        actual_value = foo()  # Returns [1, 2, 3] in no particular order
        self.assertListEqual(expected_value, actual_value)

In the above scenario, our test case will be flaky as assertListEqual is designed to match the elements in the order of our expected value. As our test case function foo() returns the same elements in no particular order, it is more suitable to use assertCountEqual

import unittest
from my_app.utils import foo

class TestListComparison(unittest.TestCase):
    def test_assert_count_equal(self):
        expected_value = [1, 2, 3]
        actual_value = foo()  # Returns [1, 2, 3] in no particular order
        self.assertCountEqual(expected_value, actual_value)

5. Date-time Flakiness

DateTime flakiness in testing can occur due to various factors, not merely as a systemic issue. Fluctuations in system time, environment differences, and asynchronous operations are the primary reasons behind this unpredictability.

Let’s take a look at a scenario where a test case measures the elapsed time between two operations:

import datetime
import time
from unittest import TestCase, mock

def is_within_business_hours(current_time):
  """Checks if the current time falls within business hours (9am to 5pm)."""
  start_of_business = datetime.time(9, 0)
  end_of_business = datetime.time(17, 0)
  return start_of_business <= current_time <= end_of_business

class DatetimeFlakinessTestCase(TestCase):
    def test_business_hours(self):
          """Tests if the current time is within business hours."""
          current_time = datetime.datetime.now().time()
          assert is_within_business_hours(current_time)

        # This test might fail if it runs exactly at 9:00 or 5:00
        # due to the nature of datetime.now() capturing a specific moment.

This test relies on datetime.now() to get the current time. If the test runs exactly at 9:00 am or 5:00 pm, it might fail because is_within_business_hours checks for strict inequalities (<= and >=). A test run at 9:00 on the dot might capture a time slightly before 9:00, causing the test to fail.

This is a flaky test because its outcome depends on the exact moment it runs, not the functionality it tries to verify.

Although the above example may be the trivial and rather incorrect approach to writing any test case, we can still see the underlying patterns of flakiness emerge.

To address this, we can go back to the idea of mocking described earlier to address the datetime.datetime.now() function using the unittest.mock module

The updated test case would look something like this:

import datetime
import time
from unittest import TestCase, mock

def is_within_business_hours(current_time):
    """Checks if the current time falls within business hours (9am to 5pm)."""
    start_of_business = datetime.time(9, 0)
    end_of_business = datetime.time(17, 0)
    return start_of_business <= current_time <= end_of_business

class DatetimeFlakinessTestCase(TestCase):
    @patch('datetime.datetime')
    def test_business_hours(self, mock_datetime):
        """Tests if the current time is within business hours (using mock)."""
        # Set a fixed time within business hours
        mock_datetime.now.return_value = datetime.datetime(2024, 4, 1, 10, 45)
        current_time = datetime.datetime.now().time() # Now uses the mocked time
        assert is_within_business_hours(current_time)

In this modified test case, we mock datetime.datetime.now() to provide fixed timestamps for the start and end times of the operation. By controlling the timestamps, we eliminate DateTime flakiness, ensuring consistent test outcomes regardless of system time fluctuations or asynchronous operations.

What actions can we take to tackle this?

Since flaky test cases can occur for apparently any reason, there is no one cheat sheet available that can be used for all root causes. Although we can be more proactive and mindful about writing resilient test cases and ensuring thorough reviews, sometimes a flaky test case requires diving into nuances of our application’s core logic.

Some starting approaches we can take to tackle flakiness are ➖

Using tools specifically designed to address flaky test cases

It can be counterproductive to sit and rerun test case after test case waiting for something to happen. Thankfully, there are tools available that allow us to execute test cases locally in bulk so we can identify flaky test cases early and often.

For pytest, you can use pytest-flakefinder . This tool allows you to multiply your tests so they run in bulk without having to restart pytest

A good rule of thumb if using these tools is to run them with a combination of parameters to try and see which conditions cause breakage. An example would be when using pytest-xdist distributed testing alongside pytest-flakefinder

pytest src/tests/test.py --flake-finder -n4 --flake-runs=n --reuse-db
pytest src/tests/test.py --flake-finder -n0 --flake-runs=n --reuse-db
pytest src/tests/test.py --flake-finder --flake-runs=n --reuse-db
pytest src/tests/test.py -n4 --reuse-db
pytest src/tests/test.py --reuse-db

where n in range of [1,2,5,10,50,500] the number of “multiplications” we want to test for.

This approach to testing can ensure that some of the more obvious causes of flakiness are identified early and should be adopted as a practice when raising PRs.

Reproducing CI executions locally

More than likely, when you use a CI tool for automated tests, chances are you would be distributing test cases across multiple containers to speed up execution times.

As mentioned above, test cases running in isolation may not be an issue but test cases that inadvertently cause a "test-on-test dependency", and flakiness may occur.

For larger codebases, there can be a lot of tests that make identifying the “problem child” rather tedious. In cases like this, a good approach would be to inspect what test cases were run before your alleged flaky test case on your CI tool and pull in the same execution parameters locally.

Re-runs, Re-runs, and Re-runs!

Believe it or not, sometimes a good old-fashioned re-run is all it takes. Having a retry mechanism with your test cases can be beneficial to reduce failing builds for your application and improve productivity. Various tools out there can be utilized for this such as pytest-rerunfailures

While this approach may result in frequently failing builds, we should aim to limit the number of retries to avoid flaky tests to continue living on in the system. Finding the right balance of retries can vary but a good starting point that has worked for us is no more than 3.

Conclusion

Flaky test cases are elusive. More often than not they can slip through the cracks of development and reviews, causing undue frustrations for everyone else down the line. As developers, we should adopt a proactive approach to resolving flakiness in test cases.

Sometimes it may seem easier to quarantine or skip the flakes as we find them but having them end up in a bucket that no one addresses can lead to an unstable platform that hasn’t addressed its core issues.

It is important to note that there is never a one-size-fits-all solution to resolving these things and more often than not, the real root causes can be more nuanced and hint towards deeper issues within the system. However, by being mindful of the different caveats of our tests during development and being proactive and prompt with resolving flaky test cases as they creep into our CI builds, we can ensure the reliability of our systems and restore trust in our external tools.

In today's interconnected world, data isn't just about rows and columns—it's about relationships. From social networks connecting friends and family to recommendation engines suggesting our next favorite movie, understanding these intricate connections is essential for unlocking the full potential of our data.

Enter graph databases—the specialized database designed to do just that.

In this blog post, we'll delve into the world of graph databases, exploring their unique features, typical use cases, and alternatives, to answer the question: To use or not to use?

Understanding Graph Databases

If this is you, don’t worry, we’ve got you covered. Let’s dive into understanding the basics.

Fundamentals

A graph database is a specialized, single-purpose platform used to create and manipulate data of an associative and contextual nature. The graph itself contains nodes, edges, and properties that come together to allow users to represent and store data in a way that relational databases aren’t equipped to do.

The main concept of a graph database system is a relationship. Relationships are defined as first-class citizens — this means everything you can do with all other elements can be done with a relationship. Data is related together in a graph to store a collection of nodes and edges, where the edges represent the relationship between nodes.

Relationships allow data within the system to be linked together directly. Querying relationships in a graph database is fast since they’re stored in a way that doesn’t change. You may also visualize them, which makes them great for deriving insights for heavily interconnected data.

In this example: User are nodes that have properties name, surname and FOLLOWS is the relationship between the nodes which can have properties (like since ) of it’s own.

Now that we are done with the basics, let’s move on to the next segment

Use Cases of Graph Databases

1. Social Networks: Graph databases are ideal for modeling social networks, where users, friendships, likes, and interactions form a complex web of relationships. By representing users as nodes and connections between them as edges, graph databases facilitate efficient querying for friend recommendations, community detection, and content personalization.

2. Recommendation Engines: Graph databases power recommendation engines by analyzing user behavior, preferences, and item similarities to generate personalized recommendations. By modeling users, items, and their interactions as nodes and edges, graph databases enable efficient recommendation algorithms that can scale to millions of users and items.

3. Fraud Detection: Graph databases play a crucial role in fraud detection by identifying patterns of suspicious behavior within networks of transactions, users, and entities. By analyzing the flow of money and connections between accounts, graph databases can uncover fraudulent activities such as money laundering, identity theft, and insider fraud.

4. Network Analysis: Graph databases are widely used in network analysis applications, including transportation networks, telecommunications networks, and biological networks. By representing nodes as network elements and connections as edges, graph databases enable advanced analytics such as route optimization, network visualization, and gene pathway analysis.

5. Features involving Hierarchical data: In addition to the standard use cases mentioned earlier, graph databases offer flexibility for implementing custom business logic involving hierarchical relationships. Whether managing organizational charts, product categorizations, or any other nested data models, graph databases excel at representing and querying hierarchical relationships efficiently. This versatility allows businesses to implement bespoke solutions tailored to their unique requirements, empowering them to derive insights and make informed decisions based on the underlying data relationships.

Commonly used graph query languages

As graph databases gain traction, the number of graph query languages has exploded. This abundance of options can be quite daunting for developers. Navigating this ever-growing landscape can be challenging.

Let’s narrow those and take a quick look at some of the most commonly used graph query languages out there:

Cypher

Cypher is an open-source declarative query language developed by Neo4j for querying graph databases. It is part of the openCypher project, an open standard that aims to make Cypher available for use in various graph database systems, which has led to it being one of the most widely adopted query languages. Queries in Cypher are typically straightforward, beginning with the specification of a pattern to match and then permitting further refinements through filtering, aggregation, and other operations.

Gremlin

Developed as part of the Apache TinkerPop graph computing framework, Gremlin is notable for its language agnosticism, meaning it is not bound to a particular graph database system. Instead, Gremlin is compatible with various graph databases, making it a suitable choice for developers who require flexibility and want to work with multiple data sources.

GraphQL

GraphQL is a query language developed internally by Facebook in 2012 before being publicly released in 2015. It has gained significant popularity as an alternative to REST APIs. Unlike the other graph query languages discussed, GraphQL was designed for clients to query data from APIs and servers, not traditional graph databases. However, it shares similarities in its graph-structured queries.

Apart from these, there are more vendor-specific graph languages like AQL(ArrangoDB), GSQL(TigerGraph), nGQL(Nebula Graph) and many more.

Alternatives to Graph Databases

Even though graph databases excel at managing interconnected data, their implementation and maintenance can be a time-consuming and expensive endeavor. However, they're not the only game in town.

Several alternative approaches exist, each with its strengths and weaknesses. Let's explore some of these alternatives and how they stack up against graph databases.

1. Recursive Common Table Expressions (CTEs) in Relational Databases: Relational databases, such as PostgreSQL and SQL Server, offer support for recursive common table expressions (CTEs). This feature allows for recursive queries that can traverse hierarchical or graph-like structures within relational data models. By recursively joining tables with themselves, developers can perform graph traversals and pathfinding operations directly within SQL queries.

2. Caching Solutions like Redis or Memcached: Another approach to handling graph-like data involves leveraging caching solutions like Redis or Memcached. These in-memory key-value stores excel at storing and retrieving data with low latency, making them ideal for implementing algorithms like breadth-first search (BFS) on graph-like data structures. By caching intermediate results, developers can improve the performance of graph traversal operations, especially in scenarios with high read/write ratios or frequently changing data.

3. Apache Age: Apache Age is a distributed graph database built on top of PostgreSQL. It leverages the capabilities of PostgreSQL as a relational database while providing native support for graph data structures and operations. With Apache Age, users can store and query graph data using familiar SQL syntax, making it a compelling alternative for organizations already invested in PostgreSQL infrastructure.

4. pgRouting: pgRouting is an extension for PostgreSQL that adds routing functionality to the database, enabling users to perform complex routing and pathfinding operations on spatial data. While not a full-fledged graph database, pgRouting can be used to solve graph-related problems such as finding the shortest path between two points in a network. It offers a range of routing algorithms and functions, making it a valuable tool for applications requiring geospatial analysis and routing.

Trade-offs and Considerations

If you’re in this same dilemma, here are several factors to consider:

• Performance: Graph databases are optimized for querying and traversing graph-like data structures, offering efficient graph algorithms and indexing mechanisms out of the box. In contrast, recursive CTEs in relational databases may suffer from performance limitations as the size of the dataset grows, while caching solutions may introduce overhead due to cache invalidation and synchronization.

• Scalability: Graph databases are designed to scale horizontally to handle large and highly interconnected datasets. However, they may require specialized infrastructure and tuning to achieve optimal performance at scale. On the other hand, relational databases and caching solutions can also scale horizontally, but may face limitations in handling complex graph traversals efficiently.

• Ease of Use: Graph databases typically offer high-level query languages and intuitive data modeling tools tailored specifically for graph data. This makes them easy to use for developers familiar with graph concepts. In contrast, leveraging recursive CTEs or caching solutions may require more advanced SQL skills or custom code to implement and maintain.

• Pricing: Graph databases often come with pricing models that consider factors such as the volume of data stored, the number of transactions processed, and the level of support provided. While some graph databases offer community editions or open-source options with no upfront costs, others may require subscription-based licensing or usage-based pricing models. In contrast, relational databases and caching solutions may have different pricing structures, such as per-instance fees or pay-as-you-go pricing for cloud-based deployments. It's essential to consider the total cost of ownership, including licensing, infrastructure, and ongoing maintenance, when evaluating the pricing of graph databases and alternatives.

• Documentation: While graph database vendors strive to provide comprehensive documentation, the quality and coverage may vary depending on the specific vendor. In contrast, alternatives such as relational databases often offer consistently well-established documentation that covers a wide range of use cases. The extensive documentation, backed by a large community, makes it easy for users to find resources and support for their projects.

• Community Support: While graph databases have an active & growing community, they may not be as extensive as those surrounding alternatives like relational databases. Relational databases have been around for more years and are more mature, resulting in a larger and more established user base. This broad user base provides robust support, forums, and resources for users, ensuring a wealth of knowledge and assistance is readily available.

Conclusion

In the ever-evolving landscape of data management, the choice between graph databases and alternative approaches is not always straightforward. Each option offers its own set of advantages, trade-offs, and pricing considerations, making it essential to carefully evaluate your specific requirements and constraints before making a decision.

Graph databases shine in scenarios where complex relationships and graph-like data structures are prevalent, offering intuitive query languages, efficient graph traversal algorithms, and scalable infrastructure. However, it's important to note that they may come with significant upfront costs, making them a substantial investment for organizations. Additionally, utilizing graph databases effectively often requires specialized expertise, further adding to the overall cost of implementation and maintenance.

On the other hand, alternatives like recursive CTEs in relational databases and caching solutions like Redis or Memcached provide viable options for certain use cases, offering familiar SQL-based querying capabilities and low-latency data access. However, they may face performance limitations or scalability challenges when dealing with highly interconnected datasets.

When making your decision, consider factors such as performance, scalability, ease of use, and pricing, weighing the trade-offs against your specific application requirements. Whether you opt for the flexibility of graph databases, the familiarity of relational databases, or the speed of caching solutions, remember that the ultimate goal is to empower your organization with the right tools to derive insights and drive innovation from your data.

In the end, the best approach is one that aligns closely with your business objectives and enables you to extract maximum value from your data assets. So, choose wisely, and embark on your data journey with confidence, knowing that you've selected the optimal solution for your needs.

References

https://www.dylanpaulus.com/posts/postgres-is-a-graph-database/

https://www.linkedin.com/pulse/you-dont-need-graph-database-modeling-graphs-trees-viktor-qvarfordt-efzof

https://www.datacamp.com/blog/what-is-a-graph-database

https://linkurious.com/graph-query-languages/

1. Basic understanding of PixiJS

PixiJS is an open-source, web-based rendering system that provides blazing-fast performance for graphics-intensive projects.

This blog assumes the user has set up a basic Pixi.js application. If not, it's highly recommended to go through the official Pixi.js tutorials

2. SVG

SVG is an XML-based vector image format for defining two-dimensional graphics, having support for interactivity and animation.

• Basic understanding of <symbol> and <use> element

  The <symbol> element is used to define graphical template objects which can be instantiated by a <use> element. Refer to MDN for reference.

3. How to use spritesheet

In layman's terms, spritesheet is a collection of small images, like icons in this case. Instead of keeping each picture separately, you tile them on a single large document. To view an image from this spritesheet, you just need the position of the image.

⚔️ The Challenge

If you directly use SVG icons within Pixi JS, they get "Rasterized" (meaning they lose their vector data). Which means if you scale them, they get blurred out. To avoid this, we can:

1. Either convert them into native Pixi Objects using something like pixi-svg library.

   • The problem with this approach is that these conversions don't translate 1:1 to original SVG due to lack of certain features. So this option can be used if your SVG is not using any of the unsupported features listed by the library.

2. Or we can set the SVG scale beforehand so we can rasterize them at higher resolution, before they get converted to textures.

For the scope of this blog post, we will be implementing the solution based on the second approach, while making sure that we have:

• Efficient Icon Rendering Performance - Icons are loaded quickly and rendered without any hassle.

• Smoother Icon Management - Adding new icons should not require significant extra effort

🔖 The Action Plan

Let's get started, with our action plan.

1. NodeJS Script to generate Spritesheet

   1. The first step is to create a spritesheet from available SVG icons in our workspace. The advantage of this tooling is rich DX and a better way to manage SVG icons.

   2. Assumption is that we already have a set of SVG icons with us

   3. The script will read these icons and compile them into a single spritesheet.json file, which we'll use within our code.

   4. The script itself will be responsible for:

      1. loading SVG icons as strings

      2. Wrapping <symbol> around them so they become "reusable".

      3. Finally, using them by <use> and laying them horizontally. (with the assumption that every icon is of the same size and aspect ratio)

      4. Following will be format for spritesheet.json

            // Spritesheet.json
            {
                "spriteSheetSVGData": "..." // Entire SVG as string
                "iconSequence":[] // sequence of icons in the spritesheet.
            }
         

2. Using SVGs in Pixi.JS

   1. Now coming to our main logic, we will use the generated spritesheet JSON to:

   2. Create texture from the spritesheet

   3. Create a function to crop the spritesheet based on the provided iconName and return the resultant sprite.

   4. Render the icon in a PixiJS Stage.

🚀 Code Walkthroughs

🧑🏽‍💻 Spritesheet generation

• spritesheet.json consists of the following properties

  • spritesheetSVGData: stringified SVG Data of the spritesheet.

  • iconSequence : An ordered array of distinct svg-ids used in the spritesheet. This is a critical attribute, whose index will be used to fetch an Icon, which is going to be used in Pixi.js app.

      /**
       * Note:
       * The script runs in node-js environment.
       */

      import path from "path";
      import fs from "fs-extra";

      /**
       * Converts string to PascalCase
       */

      const toPascalCase = (str) => {
        return `${str}`
          .replace(/[-_]+/g, " ")
          .replace(/[^\w\s]/g, "")
          .replace(
            /\s+(.)(\w*)/g,
            ($1, $2, $3) => `${$2.toUpperCase() + $3.toLowerCase()}`,
          )
          .replace(/\w/, (s) => s.toUpperCase());
      };

      // Directory where spritesheet.json will be generated
      const spriteSheetDirectoryPath = "";

      // spritesheet object, to be loaded in JSON.
      const spriteSheet = {
        spriteSheetSVGData: "",
        iconSequence: [],
      };

      // Utility function to replace SVG with <symbol>
      const replaceSvgWithSymbol = (svg, id) => {
        // Replace <svg> opening tag with <symbol> opening tag
        let modifiedSvg = svg.replace(/<svg([^>]*)>/, `<symbol$1 id="${id}">`);
        // Replace </svg> closing tag with </symbol> closing tag
        modifiedSvg = modifiedSvg.replace("</svg>", "</symbol>");
        // Remove xmlns property from <symbol>
        modifiedSvg = modifiedSvg.replace(' xmlns="http://www.w3.org/2000/svg"', "");

        return modifiedSvg;
      };

      export const createSpriteSheet = (svgList) => {
        /* 
           Used to store <symbol>(s). 
           Every root <symbol> represents a svg icon.
           For example:
           <symbol id="iconName">...</symbol>
        */
        let symbols = "";

        // conversion of SVG to <symbol>, and
        // updating IconSequence in spritesheet
        svgList.forEach((svg) => {
          const svgCode = svg.data;
          const iconName = toPascalCase(svg.name);
          const symbolCode = replaceSvgWithSymbol(svgCode, iconName);
          symbols += symbolCode;
          spriteSheet.iconSequence.push(iconName);
        });

        /* 
           Create SpriteSheet SVG Data
           Our icons are of size: 16 X 16

           A linear spritesheet having a
           height: 16 units (based on icon height).
           width: 16 * total icons
           gap between icons: 8px

          The data part of spritesheet svg
          We use entity reference method, to put images in the spritesheet.
          <use href="symbol-id" y=0 x={index * 24}/>

          Finally write the spriteSheet to the respective JSON file

        */

        spriteSheet.spriteSheetSVGData = `<svg xmlns="http://www.w3.org/2000/svg" 
      height="16" width="${16 * svgList.length}" viewbox="0 0 ${
          16 * svgList.length
        } 16">${symbols}${spriteSheet.iconSequence
          .map((symbolID, index) => {
            return `<use href="#${symbolID}" x="${index * 24}" y="0"/>`;
          })
          .join(" ")}</svg>`;

        // Create Spritesheet.json
        fs.writeFileSync(
          path.resolve(spriteSheetDirectoryPath, "spritesheet.json"),
          JSON.stringify(spriteSheet),
        );
        // Create Spritesheet.svg
        fs.writeFileSync(
          path.resolve(spriteSheetDirectoryPath, "spritesheet.svg"),
          JSON.stringify(spriteSheet.spriteSheetSVGData),
        );
      };

⚙️ Using SVGs in Pixi.JS

1. Assumptions

   • spritesheet.svg is generated from the above script.

   • iconSequence is extracted from the spriteSheet.json

2. Example App Structure

   1. We will be using Vanilla Javascript as an example for simplicity and easier for understanding

          app/
           ├─ index.html
           │─ js/
           │  ├─ app.js
           ├─ static/
           │  ├─spritesheet.json
      

3. As we are using vanilla javascript. We will be using the native DOM to load the Pixi.JS. Henceforth, here is what index.html looks like.

   • Note: The script, app.js is loaded as a module. This is intentionally done to prevent scope ambiguity issues with global context. Henceforth, we have to explicitly mention what variables we want to add to the global context. For example: globalThis.__PIXI_APP__ = app;

    <!--index.html-->
    <!DOCTYPE html>
    <html lang="en">
      <head>
        <meta charset="UTF-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
        <meta http-equiv="X-UA-Compatible" content="ie=edge" />
        <title>PixiJS and Icons Integration</title>
      </head>
      <body>
        <script src="https://cdn.jsdelivr.net/npm/pixi.js@6.x/dist/browser/pixi.min.js"></script>
        <script type="module" src="js/app.js"></script>
      </body>
    </html>

1. app.js performs the following operations.

   1. Initialize PixiJS

   2. Load SVG Resource for spritesheet

   3. Create texture from the SVG resource.

   4. A function that takes input as icon name, and returns the relevant icon sprite.

   5. Create a sprite from the texture.

   6. Crop the sprite generated, based on the required index found in iconSequence

   7. If no such icon exists, return undefined.

       // App.js
       // File where our svg is stored.
       const SVG_URL = "/images/spriteSheet.svg";
       const ICON_SEQUENCE = ["House", "Tasks", "Chart"];
      
       // Initializing the PIXIJS
       let app = new PIXI.Application({
         width: 400,
         height: 300,
         resolution: window.devicePixelRatio,
         antialias: true,
         backgroundColor: 0xeeeeee,
         autoDensity: true
       });
       globalThis.__PIXI_APP__ = app;
       document.body.appendChild(app.view);
      
       // Created SVG Resource from the SVG
       const svgRes = new PIXI.SVGResource(SVG_URL, { scale: 3 });
      
       // Created Texture from the SVG Resource
       const texture = PIXI.Texture.from(svgRes);
      
       /* Returns the relevant Icon Sprite
        * @param iconName: string
        * @param options: {x:number, y:number, height:number, width:number}
        */
       function getIcon(iconName, options) {
         const sprite = new PIXI.Sprite(texture);
      
         const iconIndex = ICON_SEQUENCE.findIndex(
                           (iName) => iName === iconName);
      
         if (iconIndex < 0) {
           console.error("No icon found with name:", iconName);
           return undefined;
         }
      
         texture.on("update", () => {
           if (texture.valid) {
             texture.frame = new PIXI.Rectangle(70 * iconIndex, 0, 50, 48);
             sprite.width = options?.width || 16;
             sprite.height = options?.height || 16;
             sprite.x = options?.x || 175;
             sprite.y = options?.y || 125;
           }
         });
      
         return sprite;
       }
       const iconSprite = getIcon("Tasks", { height: 30, width: 30, x: 25, y: 25 });
       app.stage.addChild(iconSprite);
      

👨🏼‍💻 CodeSandbox

Here's a live example of our above implementation.

• https://codesandbox.io/s/icons-integration-with-pixi-js-t55fdn?file=/js/app.js

Conclusion

So let's look at the aspects we discussed before starting the blog.

1. Icon Rendering Performance

   • Now, we do not calculate textures for every icon. Instead, we calculate it once for the spritesheet, and then we crop the spritesheet based on the icon we need.

2. Icon Management

   • Spritesheets can be now easily generated and updated with new icons from our new script.

   • And every generated spritesheet, can be versioned via git.

   • In the end, the whole spritesheet is just a big SVG file, and we need not worry about having multiple files for multiple SVG icons, providing a smoother icon management experience, without any overhead of fetching individual resources.

I hope you found the blog helpful and got to learn something new today. Plus give a read to our detailed blog on building a layout engine in Pixi.JS here

We are hiring! 🎉

Solving challenging problems at scale in a fully remote team interests you, head to our careers page and apply for the position of your liking!

1. What is an iframe?

2. What are cookies?

3. What is a third-party cookie?

Introduction

Have you ever wondered why some websites ask for storage permissions or why some features don't work in certain browsers? Let's dive into the intricacies of third-party cookie restrictions in Safari and how we tackled them!

Safari, Apple's flagship browser, has always been a pioneer in the quest for user privacy and security. In recent years, it has made significant strides in protecting its users from the prying eyes of online trackers by restricting third-party cookies.

In the following discussion, we will delve into a specific challenge encountered within this privacy-focused paradigm. This blog aims to unravel the complexities of seeking storage permission in Safari, particularly when our content is loaded within an iframe and relies on third-party cookies.

The predicament arose as we endeavored to integrate our React-based web app into an iframe hosted on a different domain. This intersection of domains posed a unique challenge, prompting us to explore innovative solutions to seamlessly navigate the intricacies of Safari's third-party cookie restrictions.

Problem

Our application uses cookies for authentication. In one of the use cases, it was being rendered inside an iframe. The parent HTML document that was rendering the iframe was being hosted on an entirely different domain.

In other browsers, our application within the iframe was able to access the cookies but not in Safari. Safari uses Intelligent Tracking Prevention(ITP) to control the access of third-party cookies.

ITP aims to prevent third-party cookies, making them inaccessible in iframes unless certain conditions are met. These conditions can be found in the Webkit's official announcement.

Storage Access APIs

As per ITP

"Third-party cookie access can only be granted through the Storage Access API."

Let's look at parts of this API that concern our problem:

1. document.hasStorageAccess API Doc

   This API is used to check cookie storage access. This will return false for third-party cookies in the case of the Safari browser.

2. document.requestStorageAccess API Doc

   This API is used to ask for third-party storage(cookie) access explicitly from the user.

Both of the above APIs are available in Safari as well as in other browsers.

Webkit's official documentation explains the steps to use these APIs & the rest of the user flow(which is the basis for the following solution). We recommend giving it a read before moving ahead with this post.

Solution

The solution described below is not the only one but will help you in designing solutions for your use cases. You can also design a solution as per your needs by following the guide mentioned here.

We are using react so the above-mentioned solution is written in the concepts of react.

1. We have created separate utility functions in the helper file.

    export function isSafari(): boolean{
      const userAgent = navigator.userAgent.toLowerCase();
      return (
        userAgent.indexOf("safari") !== -1 && userAgent.indexOf("chrome") === -1
      );
    };
   
    function supportStorageAccessApi(): boolean {
      return "hasStorageAccess" in document && "requestStorageAccess" in document;
    }
   
    export function hasStorageAccess(): Promise<boolean> {
      return document.hasStorageAccess();
    }
   
    export function requestStorageAccess(): Promise<void> {
      return document.requestStorageAccess();
    }
   
    export function requiresStoragePermissions(): boolean {
      return isSafari() && supportStorageAccessApi();
    }
   

   The above code is to make the browser API abstract from the actual implementation. These functions are what we are going to call.

2. Then after, we created a new file named useStoragePermissions.tsx and added the below-mentioned code.

    export const useStoragePermissions = (): {
      needPermission: boolean;
      askForPermission: () => void;
      haveCheckedPermission: boolean;
    } => {
      const [needPermission, setNeedPermission] = React.useState(
        requiresStoragePermissions() ? true : false
      );
      const [haveCheckedPermission, setHaveCheckedPermission] =
        React.useState(false);
   
      const isHavingPermissionFn = useCallback(async () => {
        try {
          return await hasStorageAccess();
        } catch (e: any) {
          // Handle error gracefully and show user some message
          return false;
        }
      }, []);
   
      const checkPermission = useCallback(() => {
        isHavingPermissionFn().then((isHavingPerm: boolean) => {
          setNeedPermission(!isHavingPerm);
          setHaveCheckedPermission(true);
        });
      }, [isHavingPermissionFn]);
   
      const askForPermission = useCallback(async () => {
        try {
          await requestStorageAccess();
          checkPermission();
        } catch (e: any) {
          // Handle error gracefully and show user some message
        }
      }, [checkPermission]);
   
      React.useEffect(() => {
        if (requiresStoragePermissions()) {
          checkPermission();
        }
      }, [checkPermission]);
   
      return {
        needPermission,
        askForPermission: requiresStoragePermissions()
          ? askForPermission
          : () => {},
        haveCheckedPermission
      };
    };
   

   Using the above hook we have exposed below three states:

   1. needPermission: This will be true when the browser is Safari and it has support for hasStorageAccess and requestStroageAccess

      Hint: Use this boolean while consuming this hook to decide when to call askForPermission and haveCheckedPermission .

   2. askForPermission: This is the function that the consumer could call to request the user to give storage access permission

   3. haveCheckedPermission : This is a boolean which will be true after calling askForPermission in the case of needPermission is true initially.

3. To consume the above hook, we have followed the below-mentioned steps:

   1. We have mounted and created a hook in #2 at the initialization part of the app. Use the needPermission state from it and proceed ahead as normal when needPermission is false.

   2. We created some other routes user-access-flow that show a button with some text like Set Cookie . And onClick of it, we set the cookie.

      1. This is where we are actually calling authentication-related APIs and then the server is setting up cookies.

      2. Once this cookie is set, close this tab using window.close()

   3. Now when, needPermission is true

      1. We show the user some text like, "Click here and click on the Set-Cookie button on the newly opened tab" and on click of it, redirect the user to the route created in above step 2.

      2. Now, when a user comes back from that route, call askForPermission on click of some button. Which should ask the user to give storage permission.

         Ex:

         

      3. Once the user clicks on Allow the button, your website is authorized to store and access third-party cookies and now you can continue with business logic.

      4. We have also kept this thing in mind that this consent will be revoked if the user cleans up the browser history and does not visit that domain for 7 days. These constraints are already mentioned here

Conclusion

If you have faced such issues while developing or browsing such issues, please share those in the comments. We will be more than happy to read and comment more on those.

We are hiring!

If solving challenging problems at scale in a fully remote team interests you, head to our careers page and apply for the position of your liking!

• You put two div together, they get stacked automatically.

• They grow in size as you add content to them.

• You can style them, change alignments and whatnot.

The logic that's handling all this is your browser's layout engine (along with a lot of other things).

But when we're using WebGL, everything has to be individually positioned and their dimensions have to be predefined as well. So, if we want to get an HTML-like experience within WebGL, that is, relative positioning between objects; we require a layout engine. A layout engine will help in dynamically allocating and calculating the position and dimensions of relatively positioned objects within WebGL.

What is a Layout Engine?

In this context, we're referring to a piece of logic that is responsible for handling the logistics of where something should be positioned, relative to the position of others.

Even this practical guide is using WebGL as a most common use case but the approach is not specific to WebGL. It can be used in other use cases with similar requirements.

Option 1 - Find something that "just works"

Your best bet is to find something purpose-built for this exact use case. How well it integrates with your specific project, is another thing.

Here are a couple of options:

• Yoga layout - https://github.com/facebook/yoga (C++)

  This is what React Native uses internally. That's how you can write CSS-like styles and they work nearly identically on iOS and Android.

• Stretch layout - https://github.com/vislyhq/stretch (Rust)

• Pixi Layout - https://github.com/pixijs/layout (JS)

• And a few others (which have not been updated for a while)

  • https://github.com/lynaghk/subform-layout (Abandoned, also only minified code is available)

  • https://github.com/randrew/layout (C++)

Unfortunately, for our use case (and without getting into details), none of those worked as we expected.

Maybe you're in the same situation as us, or you just want something small and don't want to use these big libraries, either way, I hope you will find something useful in this article.

Option 2 - Learn how a layout engine works

There's no point in putting these details here, especially because Matt Brubeck has done a far better job at highlighting all the important bits that someone building a layout engine should know.

Here's the link - https://limpet.net/mbrubeck/2014/08/08/toy-layout-engine-1.html

While his approach was more towards building an HTML rendering engine, the core design of the layout engine stays more or less the same.

Building a layout engine in Javascript

Please note that this is not a one-size-fits-all solution. This is more of a "proof-of-concept" than anything else. Here, we're merely trying to highlight the process that goes into building a very basic layout engine, so that one can learn from it and build their implementation, for their specific use-case. You can make this as simple as needed or as complicated as needed.

1. Plan

Before we start the implementation, let's just go over what we plan to achieve and how we plan to achieve it.

The process from start to finish goes somewhat like this:

1. Define - Have all the nodes with styles in them, arranged in a proper hierarchy. This will be the tree structure and the top-mode node will be called the rootNode

2. Compute - Start computing layouts from the rootNode, and recursively calculate for the entire tree.

3. Paint - Start from the rootNode and recursively go over all children.

2. Implementation

These are the items we'll be implementing here

• Allocator (class)

  An Allocator is a singleton, responsible for handling all the operations of adding/removing nodes.

• Box (class)

  The Box will be a visual element. It'll just use the calculated data from the node to position and shape itself accordingly.

• Test environment

  To test everything.

• Layout calculator (function)

  A recursive function that will calculate the layout for the given node (and its children) recursively.

2.1 - Allocator

2.1.1 - Define the shape of Node

Nodes are a logical structure that we will use to represent any element during the entire process. Node doesn't need to know about what kind of element it is representing. An element can be whatever you want it to be - Box, Text, anything. But they all still have to be linked to their own individual "node" which will represent them during the layout process.

A node can contain information like:

• Externally supplied style to this element

• Layout parameters of the element

For our use case, let's say that the node is defined by the following TypeScript types:

// layoutEngine.ts

type NodeLayout = {
  id: string; // unique ID
  width: number; // computed
  height: number; // computed
  x: number; // computed
  y: number; // computed
  style: NodeStyle; // externally supplied styles
  children: Array<NodeLayout>; // in order
  parentNode: NodeLayout | null; // because root will have `null`
}

type NodeStyle = {
  width?: number; 
  height?: number;
  layoutMode: LayoutMode
}

enum LayoutMode {
  HORIZONTAL = "horizontal",
  VERTICAL = "vertical"
}

2.1.2 - Create the Allocator

An Allocator singleton class will be responsible for managing all the operations of adding nodes to other nodes and forming a tree.

// allocator.ts

class Allocator {
  public rootNode = createNode(); // Wait what?
  // .. other methods will go here
}

Oh yes, we also need a function that will help us create a node initializer. Nothing too complicated.

// allocator.ts

const defaultStyle: NodeStyle = {
  layoutMode: LayoutMode.VERTICAL,
  backgroundColor: 0x000000
};

/**
 * Creates and returns a NodeLayout object with 
 * default values assigned to it.
 */
export const createNode = () => {
  const newNode: NodeLayout = {
    id: uuid(),
    width: 0,
    height: 0,
    x: 0,
    y: 0,
    style: defaultStyle,
    children: [],
    parentNode: null
  };
  return newNode;
};

Now that, we've taken care of that, let's get back to our singleton.

Now we need a way to:

• Attach new nodes to existing nodes

    attachChild(child: NodeLayout, parent?: NodeLayout) {
      const finalParent = parent || this.rootNode;
      finalParent.children.push(child);
      child.parentNode = finalParent;
    }
  

• Detach nodes from existing nodes

    detachChild(child: NodeLayout) {
      child.parentNode.children = child.parentNode.children.filter(
        (c) => c.id !== child.id
      );
      child.parentNode = null;
    }
  

• Move nodes between nodes

    moveChild(child: NodeLayout, parent: NodeLayout) {
      this.detachChild(child);
      this.attachChild(child, parent);
    }
  

• and Destroy the nodes, recursively.

    destroy(node: NodeLayout) {
      node.children.forEach((child) => this.destroy(child));
      this.detachChild(node);
    }
  

That's it. Now the only thing left to do is, actually instantiate it as a singleton and export it.

// allocator.ts

export const defaultAllocator = new Allocator();

2.2 - Box, the visual component

Again, this is just a POC, so we'll go as simple as possible. For any element, in this case, Box, we'll just have a class that will contain an instance of a NodeLayout, and then the rest of the implementation is only around how the box renders and everything else involved with it.

For the sake of our example, we're going with Pixi JS to create this box. But this can be whatever else it needs to be. The important part is how we consume the data of the node to render the box.

// Box.ts

import { createNode } from "./allocator";
import { NodeLayout, NodeStyle } from "./layoutEngine";
import * as PIXI from "pixi.js";

export class Box {
  node: NodeLayout = createNode();
  container: PIXI.Graphics = new PIXI.Graphics();

  constructor(style?: Partial<NodeStyle>) {
    this.node.style = { ...this.node.style, ...style };
  }

  render() {
    this.container.clear();
    this.container.beginFill(this.node.style.backgroundColor);

    this.container.lineStyle(1, 0x000000);
    this.container.drawRect(
      this.node.x,
      this.node.y,
      Number(this.node.width),
      Number(this.node.height)
    );

    this.container.endFill();
  }
}

2.3 - Sandbox

Just to check if everything is in order so far, we'll just create a sandbox environment to test things out. We'll be using Pixi JS here.

// app.ts or index.ts

import * as PIXI from "pixi.js";
import { defaultAllocator } from "./allocator";
import { Box } from "./Box";
import { calculateLayout, LayoutMode } from "./layoutEngine";

function initApp(container: HTMLElement) {
  const app = new PIXI.Application({
    resizeTo: container
  });

  container.appendChild((app.view as unknown) as HTMLCanvasElement);

  //// Sample code Start ////

  // initialize a Box with Red background
  const b1 = new Box({
    backgroundColor: 0xff0000
  });

  // we need to add the PIXI component to the main stage
  app.stage.addChild(b1.container);

  // And let's just assign some values to node
  // these values will actually be computed automatically
  // when we have calculateLayout in place
  b1.node = {
    ...b1.node,
    x: 20,
    y: 20,
    width: 100,
    height: 100
  };

  // finally call the render method to pain the box
  b1.render();
  //// Sample code End ////

}

const htmlContainer = document.getElementById("app");
if (htmlContainer) {
  initApp(htmlContainer);
}

At this stage, you should get something like this:

Yay, a red rectangle on the screen.

The setup is all complete and only the last piece remains, which is what binds this whole thing together (and the whole point of this article). It was necessary to have everything else in place because layout computation is something that you'd likely want to experiment around and experimentation is fun when you can visually see the change happening as you do it.

2.4 - Layout computation

When we talk about layout engines, it mostly comes down to calculating the x, y, width and height of a node. It sounds simple enough but the key is doing calculations in a particular sequence to get it right.

Layout calculation process and implementation

1. Computation starts at the given node (rootNode for the very first iteration)

    // layoutEngine.ts
   
    function calculateLayout(node: NodeLayout) {
      const parent = node.parentNode;
      // what now?
    }
   

2. Calculate width, x and y of this node

   1. Let's check if the parent is not there. This means this is likely the root node. so we just define some variables as defaults for the root node.

         ...
         if (!parent) {
           // this is a root node
           node.width = originLayout.width;
           node.x = originLayout.x;
           node.y = originLayout.y;
         }
      

   2. If a parent is present, then we need to check for whether the parent is laying the children Horizontally or Vertically.

       ...
       else {
           const currentNodeIndex = parent.children.findIndex(
             (n) => n.id === node.id
           );
      
           // Somethings are dictated by previous sibling,
           // we let's have it ready
           const previousSibling =
             currentNodeIndex > 0
               ? parent.children[currentNodeIndex - 1]
               : null;
      
           // If the parent is laying out components vertically
           if (parent.style.layoutMode === LayoutMode.VERTICAL) {
             node.width = parent.width; // deduct padding/margin here
             node.x = parent.x; // add padding/margin here
      
             const siblingBottom = previousSibling
               ? previousSibling.y + previousSibling.height
               : 0;
      
             if (siblingBottom && previousSibling) {
               // if sibling starts at 0, ends at 200,
               // this starts at 201.
               node.y = siblingBottom + 1; 
               // Also consider adding margins here.
             } else {
               node.y = parent.y; // add margins here
             }
           }
      
           // If the parent is layout out components horizontally
           else {
             const availableWidth = parent.width; // deduct paddings
      
             // we're going lazy here but ideally you can have the
             // logic to propotionally divide the available width among
             // the children.
             node.width = availableWidth / parent.children.length;
      
             node.y = parent.y; // Addings padding/margins here
      
             const siblingLeft = previousSibling
               ? previousSibling.x + previousSibling.width
               : 0;
      
             if (previousSibling && siblingLeft) {
               node.x = siblingLeft + 1; // add margins
             } else {
               node.x = parent.x; // add padding/margins here
             }
           }
         }
      

   3. Finally, let's just set the width equal to that from the style, if it is configured

         if (node.style.width) {
           node.width = node.style.width;
         }
      

3. Start calculating the layout for the children, recursively

   (i.e. start from #1 of this process)

    node.children.forEach(calculateLayout);
   

4. Calculate height of this node
   (This is done after computing the layout for the children because height is affected by the children)

    // Case 1: When height is already defined
      if (node.style.height) {
        node.height = node.style.height;
      }
      // Case 2: If the current node is laying the children vertically
      else if (node.style.layoutMode === LayoutMode.VERTICAL) {
        // then the total height is just the total height of all
        // the children (and their margins/paddings)
   
        /**
         * get the maximum height by computing difference between y of
         * first node and the y of last node, and add the height of
         * the last node.
         * Also add the padding of the parent node and margins of
         * first and last child
         */
   
        if (node.children.length >= 1) {
          const firstChild = node.children[0];
          const lastChild = node.children[node.children.length - 1];
   
          node.height = lastChild.y + lastChild.height - firstChild.y;
        } else {
          node.height = 0;
        }
      }
      // Case 3: If the current node is laying children horizontally
      else {
        // Just find the node that has the largest height
        node.height = Math.max(...node.children.map((c) => c.height));
      }
   

5. Done

2.5 - Let's try it

To test this out, we just need to go back to the sandbox environment and set up a set of Box components.

For this test, we'll do something like this

b1 (default vertical mode)
|- b11 (horizontal mode) - Yellow
|  |- b111 - Red
|  |- b112 - Orange
|  |- b113 - Magenta
|- b12 (vertical mode) - Aqua
|  |- b121 - Green
|  |- b122 - Dark green
|  |- b123 - Dark blue

To implement this, let's go back to our sandbox, replace all the sample code and implement this as follows:

1. Add the box b1

    const b1 = new Box();
   

2. Add 2 boxes b11 and b12

     const b11 = new Box({
        layoutMode: LayoutMode.HORIZONTAL,
        backgroundColor: 0xffff00 // yellow
      });
   
      const b12 = new Box({
        backgroundColor: 0x00ffff // aqua
      });
   

3. Add 6 boxes (3 for b11 and 3 for b12)

   
    // FOR b11
      const b111 = new Box({
        width: 100,
        height: 100,
        backgroundColor: 0xff0000 // red
      });
      const b112 = new Box({
        width: 120,
        height: 120,
        backgroundColor: 0xffaa00 // orange
      });
      const b113 = new Box({
        width: 80,
        height: 80,
        backgroundColor: 0xcc00ff // magenta
      });
   
    // FOR b12
      const b121 = new Box({
        height: 100,
        backgroundColor: 0x00ff00 // green
      });
      const b122 = new Box({
        width: 200,
        height: 120,
        backgroundColor: 0x00cc00 // dark green
      });
      const b123 = new Box({
        height: 80,
        backgroundColor: 0x0000cc // dark blue
      });
   

4. Put everything within the allocator so that it forms a tree

      // Add b1 to the root
      defaultAllocator.attachChild(b1.node, defaultAllocator.rootNode);
   
      // Add b11 and b12 as b1's children
      defaultAllocator.attachChild(b11.node, b1.node);
      defaultAllocator.attachChild(b12.node, b1.node);
   
      // add 3 children b111, b112, b113 within b11
      defaultAllocator.attachChild(b111.node, b11.node);
      defaultAllocator.attachChild(b112.node, b11.node);
      defaultAllocator.attachChild(b113.node, b11.node);
   
      // add 3 children b121, b122, b123 within b12
      defaultAllocator.attachChild(b121.node, b12.node);
      defaultAllocator.attachChild(b122.node, b12.node);
      defaultAllocator.attachChild(b123.node, b12.node);
   

5. Create an array of all the elements and just attach them to the Pixi Stage.
   Still, nothing will be rendered at this point.

    const components = [            b1,
                          b11,               b12,
                    b111, b112, b113,  b121, b122, b123
    ];
   
    components.forEach((c) => app.stage.addChild(c.container));
   

6. Compute the layout and render the nodes

    calculateLayout(defaultAllocator.rootNode);
   
    components.forEach((c) => {
      c.render();
    });
   

7. Done. If everything so far was correctly done, you should see something like this.

   

Wait ... it's not over

The article was more aimed at keeping things as simple as possible to understand the basics of a layout engine. But there's no need to stop here. This can be taken as an opportunity to challenge yourself and implement more things on top of this implementation, such as:

• Alignment (both horizontal and vertical)

• Paddings and Margins

• Flexible widths of children based on proportions

CodeSandbox

This entire example is also available on CodeSandbox if you'd like to fork it there and experiment with it - https://codesandbox.io/s/layout-engine-demo-3y3hjf

Join us

We're always looking to expand and welcome talented members to our team. And the best part, it's all remote! You can work from wherever you are. Head over to our careers page and apply to any of the available positions that seem right for you.

Every web application makes use of forms at some point as they are vital for information gathering. These forms can get very large and complex depending on your use case. Ideally, we want to be able to create forms in the simplest way possible.

🚁 The Problem

The React ecosystem already has some very popular libraries to manage forms such as React Hook Form, React Final Form, Formik, and many more. All these libraries provide validation, conditional logic, form submission handling, and everything you need to create complex forms. Although, all of them require a fair amount of JSX to be written which becomes verbose and repetitive, especially with long forms and multiple sections. If your forms are large, they often need to be divided into multiple React component files for better maintainability. I think we can all agree that doing so will be very tedious and time-consuming.

🚀 The Solution

Wouldn't it be great to abstract away repetitive and verbose JSX code required to render forms? Just think about it, forms do very standard and predictable tasks. Generally, we need the following:

• Validation - Is a field required or not, Regex, etc? What error message to display if validation fails?

• Layout - Whether to place fields horizontally or vertically. Nesting fields under collapse sections.

• Conditions/Dependencies - Whether to hide or show a field/section when a certain condition is true. Deriving field A value from field B.

• Form state - Providing the initial state of the form. Perform an action on submitting.

• Field metadata - Name, label and description of the field. Component to render (TextField, NumberField, Switch, custom component..).

Creating a form should be as simple as defining the structure and properties of the form in JSON format and passing that to a React component that will render the form with all the complex validation, layout, and conditions for us. Data Driven Forms is a React library that enables us to do just that.

Data Driven Forms, as the name suggests, creates React forms using JSON data. Once we set up the library in our React app, all we need to do is define a JSON definition to create a unique form with its validation, layout, fields, conditions, and structure. It can be thought of as a declarative way to build forms. We don't need to specify how to build the form, but only what the user should see and how it should behave.

Data Driven Forms library internally makes use of React Final Form for managing the state of the form state. Although this dependency is likely to be removed in the next version of the library.

⭐️ Benefits of Data Driven approach

• Build new forms significantly faster

• Less source code

• More readable and easy to tell what the form is doing

• A well-structured and consistent way to create forms

In the future if you choose to adopt a different framework like Svelte, the JSON form definitions can still be used. You would however need to create a component that would be able to render the JSON definition in the new framework as Data Driven Form library only has support for React at the time of writing this blog.

Here is an example of how a JSON definition/schema is structured.

const exampleForm = {
    fields: [
        {
              name: "firstName",
              component: "text-field",
              label: "First Name",
              validate: ...,
              condition: ...,
        },
        ...
    ]
}

🧑🏼‍💻 Using Data Driven Forms

Here is a CodeSandbox for you to view the source code and run it yourself:

Install library

To get started, install the required npm package.

yarn add @data-driven-forms/react-form-renderer

The @data-driven-forms/react-form-renderer package contains the FormRenderer component that is responsible for rendering the form based on the schema it is passed.

We'll also need a component mapper. The @data-driven-forms/ant-component-mapper provides a Component Mapper that maps string literals to Ant Design components. The mapper is passed to FormRenderer as a prop.

yarn add @data-driven-forms/ant-component-mapper antd

You could install one of the other component mapper available too or create your own custom mapper which would allow you to make use of React components already available in your project. At Certa, we have mapped our design system components to build a custom mapper.

FormRenderer

The FormRender is a React component that contains all the logic for rendering a React form according to the schema and the configuration provided to it via props.

// SchemaForm.jsx
import FormRenderer from "@data-driven-forms/react-form-renderer/form-renderer";
import FormTemplate from "@data-driven-forms/ant-component-mapper/form-template";
import componentMapper from "@data-driven-forms/ant-component-mapper/component-mapper";
import { schema } from "./schema";

export const SchemaForm = () => {
  return (
    <FormRenderer
      schema={schema}
      componentMapper={componentMapper}
      FormTemplate={FormTemplate}
      initialValues={{}}
      onSubmit={(values) => console.log(values)}
    />
  );
};

The schema property takes the form JSON definition.

The FormTemplate property defines a template of the form. This is a component that

The initialValues property allows you to set the initial values of the fields in the form.

All the available props can be viewed here.

📜 JSON Definition

Here is what a typical form schema might look like.

// schema.js
export const schema = {
  fields: [
    {
      component: componentTypes.TEXT_FIELD,
      name: "name",
      label: "Your name",
      isRequired: true,
      validate: [{ type: validatorTypes.REQUIRED }]
    },
    {
      component: componentTypes.TEXT_FIELD,
      name: "email",
      label: "Email",
      isRequired: true,
      validate: [
        {
          type: validatorTypes.PATTERN,
          pattern: "[a-z0-9._%+-]+@[a-z0-9.-]+.[a-z]{2,}$",
          message: "Not valid email"
        }
      ]
    },
    {
      component: componentTypes.TEXT_FIELD,
      name: "confirm-email",
      label: "Confirm email",
      type: "email",
      isRequired: true,
      validate: [{ type: "same-email" }]
    },
    {
      component: componentTypes.TEXT_FIELD,
      name: "address.street",
      label: "Street"
    },
    {
      component: componentTypes.SELECT,
      name: "address.state",
      label: "State",
      options: [
        { label: "Delhi", value: "delhi" },
        { label: "Goa", value: "goa" },
        { label: "Maharashtra", value: "maharashtra" }
      ]
    },
    {
      component: componentTypes.CHECKBOX,
      name: "newsletters",
      label: "I want to receive newsletter"
    }
  ]
};

The name property defines the path where the field data needs to be mapped.

The component property defines the component to be used to render the field from the componentMapper.

The label property is just the label text of that field.

The validate property defines the validation parameters on the field. Regex is also supported.

Depending on the component property, you might need to pass in more properties. For example, the select component also needs dropdown options.

🎛 Conditions

Forms fields might need to conditionally be hidden or visible if they satisfy a certain condition. With Data Driven Forms you can easily declare this inside the field schema.

Let's take an example to show visibility constraint in action.

{
    name: "whereInGoa",
    component: "text-field",
    label: "Where in Goa?",
    condition: {
        when: "state",
        is: "goa"
    }
}

Initially this field would be hidden. Only when the field with name "state" has a value of "goa", would this field be visible.

This was a simple example. Data Driven Forms allows for more complex conditions where you can define multiple rules in not, and, or or logic. You can even change the value of the field if the condition satisfies.

🛡 Validators

Validation are important to prevent users from submitting incorrect data.

A common way to validate a field is to use regex. You can use the "pattern" validator to achieve this.

{
    name: "foo",
    ...
    validate: [
        {
            type: validatorTypes.PATTERN,
            pattern: /^Foo$/i,
            message: "This field doesn't match the required format"
        },
    ]
}

message property specifies a custom error message for when the validation fails.

Following are all the validators provided by the library:

const validatorTypes = {
  REQUIRED: 'required';
  MIN_LENGTH: 'min-length';
  MAX_LENGTH: 'max-length';
  EXACT_LENGTH: 'exact-length';
  MIN_ITEMS: 'min-items';
  MIN_NUMBER_VALUE: 'min-number-value';
  MAX_NUMBER_VALUE: 'max-number-value';
  PATTERN: 'pattern'; // regex
  URL: 'url';
}

You can define your custom validator too which is essentially a function that receives the value of the field as the first argument and is expected to return the error message string if validation fails or undefined if the validation passes.

{
    name: "foo",
    ...
    validate:  [(value) => (value === "Pawan" ? '"Pawan is not valid" : undefined)]
}

🗃 Custom Component Mapper

Using a predefined component mapper could be a great way to get started, but at some point, we want to use custom components that are already in our project.

Here is how we define a custom component mapper:

// customComponentMapper.js
import { TextField } from "./TextField";

export const customComponentMapper = {
  "text-field": TextField
};

Here the TextField component would contain the custom component prop mapping.

// TextField.jsx
import React from "react";
import { Form } from "antd";
import { useFieldApi } from "@data-driven-forms/react-form-renderer";
import { validationError } from "@data-driven-forms/ant-component-mapper";
import { Input } from "../components/Input";

export const TextField = (props) => {
  const {
    input,
    isReadOnly,
    isDisabled,
    isRequired,
    label,
    helperText,
    description,
    validateOnMount,
    meta,
    ...rest
  } = useFieldApi(props);

  const invalid = validationError(meta, validateOnMount);
  const warning = (meta.touched || validateOnMount) && meta.warning;
  const help = invalid || warning || helperText || description;

  return (
    <Form.Item
      validateStatus={!invalid ? (warning ? "warning" : "") : "error"}
      help={help}
      label={label}
      required={isRequired}
    >
      <Input
        {...input}
        onChange={(e) => input.onChange(e.target.value)}
        defaultValue={input.value}
        disabled={isDisabled || isReadOnly}
        {...rest}
      />
    </Form.Item>
  );
};

The useFieldApi hook is a wrapper around React Final Form useField hook.

In this case, Input component would be the custom component that you want to use as a text field.

If the component itself doesn't have a way to show the required symbol, label or description of a field, a wrapper component like Form.Item from ant design can be used.

We can now spread the customComponentMapper object while passing it into FormRenderer. This would override the predefined component mappers with our custom ones.

<FormRenderer
    ...
    componentMapper={{ ...componentMapper, ...customComponentMapper }}
    ...
/>

📐 FormTemplate

The available component mapper packages offer a predefined FormTemplate component to fit the design language of their respective Design Systems. At Certa, we have our own design system and needed a way to customize the button styles, placements and styling of the container form component. The best way to do this is to define our own custom FormTemplate component.

A very basic custom FormTemplate would look like this:

const FormTemplate = ({ schema, formFields }) => {
  const { handleSubmit } = useFormApi();

  return (
    <form onSubmit={handleSubmit}>
      { formFields }
      <button type="submit">Submit</button>
    </form>
  )
}

👋 Final Words

There is some initial heavy lifting required to setup Data Driven Forms, especially if you have your own design system components. But once the custom component mappers, validators, and FormTemplate are good to go, the form-building effort is significantly low and would save a lot of developer effort and time in the long run.

Hope the Data Driven approach helps you and your team focus less on forms and more on the business logic of your product.

We're hiring!

If solving challenging problems at scale in a fully-remote team interests you, head to our careers page and apply for the position of your liking!

• An existing CircleCI pipeline.

• Basic knowledge of Pytest.

• Basic knowledge of YAML files.

🌅 Background

We have a couple thousand test cases as the test suite of the Django app powering Certa. To ensure CI, we are utilizing CircleCI to validate the PRs before merging them onto the repository's development branch.

😬 The problem

Due to the reasons listed below, developers were being slowed down and had to monitor and re-run test cases when merging their PR constantly.

• The total duration of the test run before optimization was ~28 min.

• Due to the size & history of the test suite, flaky test cases used to be a frequent occurrence.

🤔 Solutions

• 📊 Distributing test cases based on timing

  • By default, test cases are distributed among containers based on the file names, which can lead to skewed timings like this:

    

    Splitting based on timings would shave out much of the test duration for no extra cost.

    Steps:

    1. Upload test run report

       After a test run to get the test cases to generate a report once their run is completed

        - run: # this will create the dir to store the reports
            name: Make test report dir
            command: mkdir -p test-results/reports
        - run: # we split the test cases using timing and generates report at the end of test run
            name: Run tests and create report
            command: |
              TESTFILES="$(circleci tests glob "<test dir path>/**/test_*.py" | circleci tests split --split-by=timings)
              pytest ${TESTFILES} --junitxml=test-results/reports/junit.xml
        - store_test_results: # this will ensure test results are stored in CircleCI for timing data for later runs.
            path: test-results
       

       Note: By default, the reports we generate use xunit2 format to store test results that do not contain the file paths and hence can't be used to generate timing data at the time of writing this blog. A workaround we used is to specify the following in the setup.cfg the file of the Django project:

        [tool:pytest]
        python_files = test*.py
        junit_family = xunit1
       

    2. Split test cases based on timings

       Once we have the timing data available on CircleCI, then we can add split-by-timing to the YAML file as follows(re-iterating):

        - run: # we split the test cases using glob and generates report at the end of test run
            name: Run tests and create report
            command: |
              TESTFILES="$(circleci tests glob "<test dir path>/**/test_*.py" | circleci tests split --split-by=timings)
              pytest ${TESTFILES} -n 4  --reuse-db --junitxml=test-results/reports/junit.xml
       

       The run command uses two things, the first is to get all the test files using glob and then to initiate the test run by specifying to use timing as the split criteria.

    3. Set timing-type

       Now that we have timing data, we can set the granularity. CircleCI supports 4 different choices: filename , classname, testname and autodetect. This choice will depend on your needs but classname worked best for our case.

  • Cost vs Impact

    • No additional cost was associated with this change, but it significantly improved the timings, reducing the overall duration by 36%.

  • Caveats

    • The store_test_results command needs a directory as input. The test results need to be stored inside this directory.

• 📈 Increasing container count

  This is an obvious option available to you, but the problem is that it will incur higher credit usage. This particular option varies based on projects, we found that 8 containers of the large class are the ideal configuration for our use case, which saves time while not burning away credits. Below are the YAML file changes:

    working_directory: ~/repo 
    resource_class: large 
    parallelism: 8 
    steps: 
      - attach_workspace: 
               at: ~/repo 
      - run: # we split the test cases using glob and generate the report at the end of the test run 
          name: Run tests and create report 
          command: | TESTFILES="$(circleci tests glob "<test dir path>/**/test_*.py" | circleci tests split --split-by=timings) pytest ${TESTFILES} -n 4 --reuse-db --junitxml=test-results/reports/junit.xml
  

  • Cost vs Impact

    • The Addon cost was around 100k credits per month. It reduced the timings by around 28% and also consumed a lot of time to find the sweet spot since each time after changing the number, we had to run the pipeline completely to get the impact.

  • Caveats

    • This will be time-consuming; what we have done to make it easier is to push several commits with only the job container count changed and run the pipelines in parallel manually. By default, it cancels a job when a new commit is pushed, so you must manually rerun the suite.

    • Increasing parallelism can also lead to an increase in flaky test cases, depending on your test suite. There can be multiple reasons for this - race conditions, resource contention, or synchronization issues. Read on to learn about the solutions that worked for us.

• 🏎️ Parallel test runs with a job

  If you already have parallel containers running the jobs, then one thing to keep track of is the resource usage section of the test run. Below is the screenshot of an unoptimized job.

  

  As the screenshot shows, jobs were utilizing at most 50% of the CPU. An easy remedy we applied is to utilize the number of parallel pytest instances running in a container. Below is the config change for increasing the instances to 4:

      working_directory: ~/repo
       resource_class: large
       parallelism: 8
       steps:
       - attach_workspace:
               at: ~/repo
       - run: # we split the test cases using glob and generate the report at the end of the test run 
           name: Run tests and create report
           command: |
             TESTFILES="$(circleci tests glob "<test dir path>/**/test_*.py" | circleci tests split --split-by=timings)
             pytest ${TESTFILES} -n 4  --reuse-db --junitxml=test-results/reports/junit.xml
  

  Note: Low CPU resource usage is expected if your test suite is IO-heavy. In such cases lowering the resource type can yield some savings on credits.

  • Cost vs Impact

    • No additional cost was associated with this change, and the impact was not great either. On average, it saved about 1-2 mins (3.8%) per test run.

  • Caveats

    • When you optimize this value, if you don't see at least a 15% improvement compared to the previous value, then it is time to stop since, post that, your cost increase will probably be unjustifiable.

• 💾 Using cache and workspace storage

  Projects that utilize external libraries or files that remain relatively constant between multiple test runs can be cached and used to save a significant amount of time. We used:

  • cache to store our python libraries by generating a key based on the hash of the requirement files

  • workspaces cache to create a common test env from which different test jobs start.

    The config is as follows:

    prep-test-env:
       parameters:
         resource_class:
           type: string
           description: CircleCI resource class
           default: medium
       docker:
           - image: cimg/python:3.9.16
       working_directory: ~/repo
       resource_class: large
       steps:
         - checkout
         - run: # Generate a hash based on requirements
             name: Check if requirements changed
             command: |
               echo $(find ./requirements -type f -exec md5sum {} \; | md5sum | cut -d' ' -f1)  >> REQUIREMENTS_CACHE_KEY
         - restore_cache:
             keys:
               - dependencies-{{ checksum "REQUIREMENTS_CACHE_KEY" }}
         - run: # this will install any new dependency added
             name: Install python dependencies
             command: |
               python -m venv venv
               . venv/bin/activate
               pip install --upgrade setuptools
               pip install -e .
               for file in requirements/*.txt; do pip install -r "$file"; done
         - save_cache:
             paths:
               - venv
             key: dependencies-{{ checksum "REQUIREMENTS_CACHE_KEY" }}
         - persist_to_workspace:
             root: ~/repo
             paths: ./
    

    In our job for the test run, we will attach this saved workspace as follows:

        steps:
        - attach_workspace:
                at: ~/repo
    

    To specify the order in which we define the following in the workflows section

      workflows:
        version: 2
        app-tests:
          jobs:
            - prep-test-env:
                name: Prepare the test environment
            - run-test:
                name: Test run
                requires:
                  - Prepare the test environment
    

    This will ensure the prep-test-env runs before the test cases are run.

  • Cost vs Impact

    • This caused an additional cost of ~3k credits per month, including caches and workspace storage. This saved about 40 seconds from each test run job, which is significant since each parallel job consumes credits independently.

  • Caveats

    • Do make sure to set the usage policy to avoid storing irrelevant data. We have set it as follows:

      

• 🔄 Auto-rerun failed test cases

  Once the optimizations were done, flaky test cases became the bane of our existence. Since this would be counter-productive to the main objective of lowering the load on developers. We utilized a library called pytest-rerunfailures, which, as the name suggests, re-runs failed test cases. But the problem is that this is not a library required for production and is not maintained/secured to be installed in production. Hence it was added in a test.txt file in the requirements directory, and since we utilize for file in requirements/*.txt; do pip install -r "$file"; done it will be installed during test runs and not in any other server.

•       - run: # we split the test cases using glob and generate the report at the end of the test run 
            name: Run tests and create report 
            command: | pytest ${TESTFILES} -n 4 --reuse-db --junitxml=test-results/reports/junit.xml --reruns 3 -x
  

  --reruns 3 reruns the failed test case up to 3 times, and if one of them passes, it goes ahead with the rest.

  -x will exit immediately after a test case fails, even after 3 reruns

  • Cost vs Impact

    • No cost add-on for this change, but it caused the flaky test cases to be reduced significantly from 2-3 flaky test cases failures per job run to 1 flaky failure in every 10-15 test runs.

  • Caveats

    • Make sure to include -x in the command, if your pipeline is configured to fail even if one test case fails since that will save some credits.

      You could also set it to only rerun test cases with a specific type of failure like this: --only-rerun AssertionError this will only rerun test cases that have failed due to AssertionError.

We're hiring!

If solving challenging problems at scale in a fully-remote team interests you, head to our careers page and apply for the position of your liking!

At Certa, our designer maintains over 150 SVG icons on a Figma board, with new ones coming every now and then. Previously we had to go through the following tedious and manual process of adding an icon from Figma to our React project:

1. Exporting the icon as SVG format from Figma
2. Creating a React component file and appropriately naming it
3. Copying and pasting the SVG code into the component file
4. Exposing props that parent component can pass (color, size, etc)
5. Modifying SVG attributes (e.g. setting fill to currentColor)
6. Clean unnecessary attributes
7. Exporting the React icon component from the index file

This was a slow, repetitive, and error-prone process. Certainly, we could write some scripts here to automate much of the process. So we did. But first, let us explore why the above process can be a problem.

🔎 The Problem

Problem 1

Everyone has their own way of performing manual work. Not every developer on the team will follow a consistent way of performing the above steps.

Let us say Alice downloaded loader.svg icon from Figma and created a component called Spinner.tsx in the React project because that name seems to make sense to Alice. Bob comes along and is searching for the loader icon by searching for Loader.tsx file assuming that icons are probably named according to their names on Figma. But Bob doesn't find the icon, so he creates a new icon component name Loader.tsx from loader.svg.

Now we have two icon components named differently, but having the same code. This will lead to code duplication unknowingly, which is never good.

Problem 2

Another issue was that of inconsistent props of icon components. In some components, certain props ( e.g. color and size) were added, and in some, it wasn't. In some components, the default value of a prop (e.g. color) was set to a hex value and in others set to currentColor. This had led to unpredictable behavior when using icons and unexpected bugs started haunting the UI.

🚀 The Plan

Our frontend codebase is a monorepo housing various packages each having its own special purpose. One such package is called blocks (@certa/blocks). It houses the design system/building block components of our application UI such as the button, menu, tooptip, etc. Previously it housed the icons too, but it didn't quite feel like the right home for it.

The proposed idea was to create a new package (@certa/icons) in the monorepo specifically catering to icons and automating as much of the icon component generation process as possible.

Maybe not all the icons you need in your project are on Figma. Not a problem, we can also load SVGs icons into the script from a specific folder as an additional source.

The new package would contain a script that would:

1. Fetch SVG icons from Figma
2. Load SVG icons added manually into a specific folder as an additional source
3. Clean and optimize the SVG code
4. Generate React components for each icon which expose common props with the same defaults
5. Create an index file that exports all the icon components

🧑‍💻 The Script

Here is a CodeSandbox for you to view the source code and run it yourself:

The script has been implemented using Node.js with TypeScript and it will make use of the following libraries:

• figma-api-exporter - for fetching icons data from Figma
• @svgr/core - transforming SVG code into React components
• ts-node - enables executing TypeScript on Node.js without precompiling
• fs-extra - adds file system methods that aren't included in the native fs module
• axios - for downloading SVGs from Figma
• dotenv - loading environment variables into the script
• chalk - printing colored output to the terminal

Firstly, we need to install the required libraries:

yarn add react figma-api-exporter fs-extra axios dotenv @svgr/core@5.5.0 @svgr/plugin-prettier@5.5.0 @svgr/plugin-svgo@5.5.0 chalk@4.1.2

yarn add --dev typescript ts-node @types/react @types/node @types/fs-extra

Folder Structure

We'll be using the directory structure given below to organize our code:

icons-generator/
├─ src/
│  ├─ icons/
│  │  ├─ components/ (all generated React icon components)
│  │  │  ├─ Bell.tsx
│  │  │  ├─ Gear.tsx
│  │  │  ├─ Home.tsx
│  │  ├─ svgs/ (manually added SVGs)
│  │  │  ├─ Gear.svg
│  │  ├─ index.tsx
│  ├─ templates/
│  ├─ generate.ts
│  ├─ types.ts
│  ├─ utils.ts
│  ├─ index.tsx
├─ .env
├─ svgr.config.js
├─ package.json

src/generate.ts file is the primary script where a majority of our logic would be.

svgr.config.js is the SVGR library configuration file

src/icons/components/ will contain all our generated react component files

src/icons/svgs/ contains SVG files that are not in Figma but also need to be converted to React components.

src/icons/index.tsx index file that exports all icon react components from components/ directory

Setup

The first step in the script would be to fetch the metadata of icons from Figma. The metadata will contain S3 URLs of all the icons which can be used to download their corresponding SVG code. We will use figma-api-exporter for this purpose.

To use this library, we would need to set up our Figma Personal Access token that the library can use to access our Figma account. Secondly, we need the Figma file id of the Figma board containing the icons. And lastly, the canvas name is required to locate the icons.

Getting Figma Personal Access Token, File ID, and Canvas

1. Go to your Figma dashboard and open Settings from the top right dropdown menu.

2. Scroll down to the "Personal access tokens" section and create an API token. Keep the API token handy, we'll be adding it to the .env file soon.

3. Now to get the canvas name, look at the sidebar on Figma. The canvas is the name of the page on a Figma board where you kept the icons. In my case, the canvas is named "Icons".

4. One more thing we need is the Figma file id. Go to the Figma board where your icons reside. Look for the file id in the URL, it'll be the 12 character alphanumeric text.

   https://www.figma.com/file/atT09besy7MsrjeyUsJLfP/Example-Board
   

5. Anyone who has your personal access token can get full access to your Figma account. It is best to save these values to a .env file so that we don't accidentally commit these values to version control. In the .env file, add the API token and the file id as key-value pairs.

    FIGMA_API_TOKEN=<YOUR_FIGMA_API_TOKEN>
    FIGMA_FILE_ID=<FIGMA_FILE_ID>
    FIGMA_CANVAS=Icons
   

6. Open the package.json file and add the following entry in the scripts section:

    "icons": "ts-node ./src/scripts/generate.ts",
   

   Now we can just use the yarn icons command to run the script.

   

Code

Enough talk, let's dive into the code.

1. Firstly, we load the environment variables.

    // generate.ts
   
    // Loads environment variables from .env
    dotenv.config();
   
    // 1. Retrieve Figma Access Token, File ID and Canvas from .env file
    const FIGMA_API_TOKEN = process.env.FIGMA_API_TOKEN;
    const FIGMA_FILE_ID = process.env.FIGMA_FILE_ID;
    const FIGMA_CANVAS = process.env.FIGMA_CANVAS;
   

2. Then we use figma-api-exporter library to fetch metadata about the icons from Figma, providing it with the token, file id and canvas name.

    // 2. Fetch icons metadata from Figma
   
    // generate.ts
    const exporter = figmaApiExporter(FIGMA_API_TOKEN);
    exporter
      .getSvgs({
        fileId: FIGMA_FILE_ID,
        canvas: FIGMA_CANVAS
      })
      .then(async svgsData => {
        console.log('SVGs DATA:', svgsData);
      })
      .catch((err: unknown) => {
        process.exit(1);
      });
   

   The console.log would output the following object structure:

    {
        "svgs": [
            {
                id: "1:43",
                url: "https://s3-us-west-2.amazonaws.com/...",
                name: "House"
            },
            {
                id: "2:86",
                url: "https://s3-us-west-2.amazonaws.com/...",
                name: "Chevron down"
            },
            ...
        ],
        "lastModified": "2021-11-30T19:19:08Z"
    }
   

   The most interesting properties here are url and name.

3. We can use the url property of each icon to download the SVG code using the axios library. A simple get request will do the trick.

    // utils.ts
    export const downloadSVGsData = async <T extends {}>(
      data: ({ url: string } & T)[]
    ) => {
      return Promise.all(
        data.map(async dataItem => {
          const downloadedSvg = await axios.get<string>(dataItem.url);
          return {
            ...dataItem,
            data: downloadedSvg.data
          };
        })
      );
    };
   
    // generate.ts
    // 3. Download SVG files from Figma
    const downloadedSVGsData = await downloadSVGsData(svgsData.svgs);
   

4. Not all your icons might be in Figma. Having a second source for adding SVG icons would be great to have. So next, we can load the manually added SVG files located in src/icons/svgs/ and combine them with the ones downloaded from Figma into a single array named allSvgs. The fs.readdirSync would come in handy, allowing us to retrieve files in a certain directory.

    // generate.ts
   
    // 4. Read manually added SVGs data
    let manuallyAddedSvgs: { data: string; name: string }[] = [];
    const svgFiles = fs
      .readdirSync(SVG_DIRECTORY_PATH)
      // Filter out hidden files (e.g. .DS_STORE)
      .filter((item) => !/(^|\/)\.[^/.]/g.test(item));
    svgFiles.forEach((fileName) => {
      const svgData = fs.readFileSync(
        path.resolve(SVG_DIRECTORY_PATH, fileName),
        "utf-8"
      );
      manuallyAddedSvgs.push({
        data: svgData,
        name: toPascalCase(fileName.replace(/svg/i, ""))
      });
    });
    const allSVGs = [...downloadedSVGsData, ...manuallyAddedSvgs];
   

5. Once we have all the SVG code, it is time to convert them into React components and write the files to the src/icons/components/ directory.

    // generate.ts
   
    // 5. Convert SVG to React Components
    allSVGs.forEach(svg => {
      const svgCode = svg.data;
      const componentName = toPascalCase(svg.name);
      const componentFileName = `${componentName}.tsx`;
   
      // Converts SVG code into React code using SVGR library
      const componentCode = svgr.sync(svgCode, svgrConfig, { componentName });
   
      // 6. Write generated component to file system
      fs.outputFileSync(
        path.resolve(ICONS_DIRECTORY_PATH, componentFileName),
        componentCode
      );
   });
   

   The svgr.sync function is performing some magic here, converting SVG code to React component. We will dive deeper into how it works in a later section.

6. After the React components are created, the process of generating an index file can begin. The index file would export all the React components, so they can be used outside our current package. Have to say, the final generated index file looks oddly satisfying. 😄

    // utils.ts
    export const createIndex = ({
      componentsDirectoryPath,
      indexDirectoryPath,
      indexFileName
    }: IndexConfigProps) => {
      let indexContent = "";
      fs.readdirSync(componentsDirectoryPath).forEach(componentFileName => {
        // Convert name to pascal case
        const componentName = toPascalCase(
          componentFileName.substr(0, componentFileName.indexOf(".")) ||
            componentFileName
        );
   
        // Compute relative path from index file to component file
        const relativePathToComponent = path.relative(
          indexDirectoryPath,
          path.resolve(componentsDirectoryPath, componentName)
        );
   
        // Export statement
        const componentExport = `export { default as ${componentName} } from "./${relativePathToComponent}";`;
   
        indexContent += componentExport + os.EOL;
      });
   
        // Write the content to file system
      fs.writeFileSync(path.resolve(indexDirectoryPath, indexFileName), indexContent);
    };
   
    // generate.ts
    // 7. Generate index.ts
    createIndex({
      componentsDirectoryPath: ICONS_DIRECTORY_PATH,
      indexDirectoryPath: INDEX_DIRECTORY_PATH,
      indexFileName: "index.tsx"
    });
   

   The above function is simply looping over all the icon components, converting the filename to a component name, computing the relative path of the component file, forming an export style string, and concatenating it to a string that then gets written to the index.tsx file.

⚙️ SVG to React Components

When it comes to raw SVG to React conversion, the SVGR library gets the job done. It provides additional plugins like SVGO that are able to shave off some excess SVG styling and help with size reduction.

I provided SVGR with a config where I customized the output React component code as per my requirements. The svgProps property is used to set the SVG attribute values. replaceAttrValues will replace an existing attribute value with a new one, very useful for changing a hexcode color for fill attribute to currentColor which essentially tells the SVG to inherit the color property value from a parent element.

// svgr.config.js
const componentTemplate = require("./src/templates/componentTemplate");

module.exports = {
  typescript: true,
  icon: true,
  svgProps: {
    width: "inherit",
    height: "inherit"
  },
  replaceAttrValues: {
    "#00164E": "currentColor"
  },
  plugins: [
    // Clean SVG files using SVGO
    "@svgr/plugin-svgo",
    // Generate JSX
    "@svgr/plugin-jsx",
    // Format the result using Prettier
    "@svgr/plugin-prettier"
  ],
  svgoConfig: {},
  template: componentTemplate
};

This is cool and all, but you might be wondering how we can customize the React component code that SVGR pushes out. Here is where templates come in. SVGR allows us to pass it a template function that is internally executed by the babel plugin babel-plugin-transform-svg-component and expects Babel AST (Abstract Syntax Tree) to be returned from the function.

AST is essentially a tree representation of program source code, and in our case represents the React component source code.

// componentTemplate.js
function componentTemplate(
  { template },
  opts,
  { imports, componentName, props, jsx, exports }
) {
  const code = `
    %%NEWLINE%%
    %%NEWLINE%%

    import * as React from 'react';
    import { IconProps } from '../../types';
    import { IconWrapper } from '../IconWrapper';

    %%NEWLINE%%

    const %%COMPONENT_NAME%% = (allProps: IconProps) => {
      const { svgProps: props, ...restProps } = allProps;
      return <IconWrapper icon={%%JSX%%} {...restProps} />
    };

    %%EXPORTS%%
  `;

  const mapping = {
    COMPONENT_NAME: componentName,
    JSX: jsx,
    EXPORTS: exports,
    NEWLINE: "\n"
  };

  /**
   * API Docs: https://babeljs.io/docs/en/babel-template#api
   */
  const typeScriptTpl = template(code, {
    plugins: ["jsx", "typescript"],
    preserveComments: true,
    syntacticPlaceholders: true
  });

  return typeScriptTpl(mapping);
}

module.exports = componentTemplate;

The template API from @babel/template is provided as part of the props. It is responsible for converting our code from a string format to AST format. We pass the mappings to the placeholders in our code when invoking the function returned by the template function.

If you've noticed, in the custom template we have an IconWrapper component that gets passed the SVG code via props. The SVG code is rendered between a span. The wrapper has the benefit of being able to modify the props' behavior in one place as opposed to keeping the logic in each generated React component.

// IconWrapper.tsx
import * as React from "react";
import { IconProps } from "../types";

export const IconWrapper: React.FC<{ icon: React.ReactNode } & IconProps> = ({
  icon,
  color: colorProp,
  size: sizeProp,
  autoSize,
  ...restProps
}) => {
  const color = colorProp ? colorProp : "currentColor";
  const size = sizeProp ? `${sizeProp}px` : autoSize ? "1em" : "16px";
  return (
    <span
      role="img"
      aria-hidden="true"
      style={{
        color: color,
        width: size,
        height: size,
        display: "inline-flex",
        fontSize: "inherit"
      }}
      {...restProps}
    >
      {icon}
    </span>
  );
};

🎨 Make it Pretty

Once the script is ready, you can improve it by adding error handling and descriptive messages. We at Certa have also abstracted away a lot of dynamic values like the index file directory, index file name, icons component directory, etc. into a separate config file to be able to easily configure changes.

If you love good design like me, you might want to use additional libraries like ora and chalk to make the developer experience awesome while waiting for the icons to be generated. Ora is what adds the cool-looking spinners to the terminal while something is loading and Chalk allows you to bring the terminal to life with colors.

Look how beautiful the final CLI tool looks:

👋 Final Words

It seems like a lot of work to get the script set up, but once it works, you gain complete control over your icons. Plus it saves a ton of time and makes your code more reliable.