A couple of weeks ago I wrote a blog post about AI and how I believe it's not up to the task of being a genuine help and support for users. You can check out that post here but I was thinking about what I'd written and wondered, was I being too harsh? Should I give AI another chance? So this blog post will be a brain dump of experience using AI tools and a review of whether things have improved since I last used them.

Using Co-Pilot(s?)

I've previously used co-pilot thanks to it being made available to me through my place of work. The user experience of using this has vastly improved since I last touched it with the biggest improvement being the chat functionality. I can clearly see the full prompt and provide my files as context to help with my development. This is particularly useful when previously I had to copy and paste the contents of a file to get any kind of help with it.

But it's still a little bit meh.

I was quite underwhelmed by the capabilities. To give credit where it's due, the ability to generate documentation (though it's sometimes wrong) is a big time saver, particularly when documenting lots of functions at the end of a long day. But when trying to solve something complex or spot an error, it's less than helpful.

My Nuxt Problem

I've been playing around with Nuxt in preparation for an upcoming project at work and have really loved working with it. But I ran into an issue that the AI could not solve no matter how many times it tried.

The issue was with testing. I was trying to set up Playwright end-to-end testing using the Vitest runner. This is well documented in the official Nuxt docs but I was running into some issues when trying it on the app the I'd made.

Issue number 1 was the following:

TypeError: Could not load `@nuxt/icon`. Is it installed?

Now this was confusing for a number of reasons that are specific to Nuxt but it boils down to how Nuxt modules work, specifically Nuxt UI. When you install and configure Nuxt UI it installs a bunch of dependency packages such as Nuxt Icon but these weren't being detected when I ran my tests.

After some searching yielded nothing useful I turned to co-pilot to solve the issue; which it did. The issue is how Nuxt's module imports work when using Vitest and Playwright together (at least I think it is but I have a feeling it could also be to do with using pnpm for my package manager) and how you have to manually install all the dependencies to make them available for testing. Co-pilot found this issue particularly quickly and, aside from the fact that it's a rubbish solution that I would like resolved, it sort it. Which was great! It had proven useful to me!

But After resolving all these issues I was still getting an error:

× Login E2E Tests > Should launch the application on the login page 485ms
    → page.goto: Protocol error (Page.navigate): Cannot navigate to invalid URL

Here was my test file at the time:

import { createPage, setup } from '@nuxt/test-utils/e2e';
import { describe, expect, it } from 'vitest';

describe('Login E2E Tests', async () => {
  await setup();

  it('Should launch the application on the login page', async () => {
    // Arrange
    page = await createPage();
    await page.goto('/', { waitUntil: 'hydration' });

    // Assert
    expect(new URL(page.url()).pathname).toBe('/login');
    expect(page.getByTestId('login-form')).toBeDefined();
  });
});

I thought to myself "it did so well on the last one, why not ask it this question and save myself some time searching for the answer myself?"

But it just couldn't solve the problem. It was coming up with all these solutions around the source directory for my project files and configuring which files to use and all sorts of wacky stuff. None of it worked. I just kept getting the same error over, and over, and over again. And each time I did what it said and told it that the issue was still present it would either ask me to do the SAME THING AGAIN or would go deeper into the config rabbit hole.

In the end, the issue was one of my own doing.

I had been following a video in the official Nuxt docs and had carelessly missed a function call:

import { createPage, setup, url } from '@nuxt/test-utils/e2e';
import { describe, expect, it } from 'vitest';

describe('Login E2E Tests', async () => {
  await setup();

  it('Should launch the application on the login page', async () => {
    // Arrange
    page = await createPage();
    await page.goto(url('/'), { waitUntil: 'hydration' });

    // Assert
    expect(new URL(page.url()).pathname).toBe('/login');
    expect(page.getByTestId('login-form')).toBeDefined();
  });
});

Can you spot the difference? It's using that newly imported url method.

Now this was a problem of my own making but co-pilot couldn't find it. It didn't have a clue. Which irked me. Isn't it meant to be the solution to these sorts of issues? These simple problems that occur because you've missed something in docs or a video?

Conclusion

Don't get me wrong, I recognise that co-pilot helped me solve an issue. But it's another showcase of where searching the internet, reading documentation and taking a break and reviewing your code is often the best solution to a problem. These are skills that will help you as a developer and engineer cultivate your own knowledge base which you can draw upon in the future.

And maybe my inate scepticism of AI plays a part in this. I don't believe that it's the solution we need to solve the issues we face in software development; particularly when working on something we're less familiar with. I find that it's at it's most useful when I already have a firm grasp of what I'm doing and the problem I'm tring to solve and it really is a co-pilot: taking on the boring tasks of writing documentation and providing suggestions of what to do, but not actually solving the complex problems that we as engineers need to be thinking about ourselves.