• MagicShel@lemmy.zip
    link
    fedilink
    English
    arrow-up
    42
    arrow-down
    1
    ·
    1 day ago

    I think this is a good article and accurate. I’ve tried to block the rising tide of this within my own company, but my management is pressing forward with plans to implement an AI harness. I pushed forward a design that at least put questions and decisions in the hands of developers, but I got my hand slapped.

    We are announcing 10x dev speeds based on pocs that haven’t been team reviewed or deployed in test. The code looks fine tbh, but I know it’s an illusion. The moment we start signing contracts based on 10x, the writing is on the fucking wall.

    AI can be an accelerator — my teams have seen 30% productivity gains without sacrificing quality, and I think I could push that higher, perhaps as much as 2x, though I’d rather just target continuous progress than predict the benchmark — but we can’t be happy with that and so we are turning an incredible success into an epic failure.

    • HaraldvonBlauzahn@feddit.org
      link
      fedilink
      arrow-up
      8
      ·
      edit-2
      7 hours ago

      I have tried that with a small side project - a simple TUI app and parser in Rust - and I wasn’t convinced it is really faster. It spat out some 4,000 lines in about four afternoons, a bit of ratatui and so on. The cursor wasn’t visible. The fix was completely hallucinated, or BS, and suggested downloading and patching another lib. The parser was incorrect. It had over one hundred dependencies. I could fix it with two or three more afternoons time. Now, 4000 lines in a few days still sounds impressive, doesn’t it?

      But then, I wanted to know it, threw the generated code away, and wrote it again from scratch. I got there with half the code and had it working. In the same time. With a working cursor and a better parser.

      I think the speed perception is like when you are using public transport or a commuter train vs a bicycle. The motorized vehicle always feels faster. But at distances like ten miles / 15 kilometers, it isn’t. A car still feels much faster, but often it isn’t worth the cost.

      These things save – seem to save – effort, which our primate brain is wired to. But laziness is a bitch. Lke cars, which when used a lot instead moving our body, only make our life shorter.

    • TheAgeOfSuperboredom@lemmy.ca
      link
      fedilink
      arrow-up
      15
      ·
      1 day ago

      Does that 30% increase include extra review time? I’m finding that my small team is producing a lot more code, but I’m also spending way more time reviewing, correcting stupid LLMisms, and keeping complexity in check. The last point is the one that concerns me the most because I just can’t keep up and complexity is starting to balloon.

      • marlowe221@lemmy.world
        link
        fedilink
        English
        arrow-up
        4
        ·
        14 hours ago

        Yeah, I’m fighting the same problem. Way too much to review, not enough time to do it, and I feel like Lucy at the chocolate factory trying to stop the ballooning complexity of certain code bases… but it doesn’t seem to matter.

        Sure, developers might be 30% more productive… in the sense that they are producing more literal lines of code in a shorter period of time. But… is that good?

        I can’t say I’m seeing a net benefit at this point on the small team I lead.

      • MagicShel@lemmy.zip
        link
        fedilink
        English
        arrow-up
        12
        ·
        1 day ago

        Yes. They are rebalancing to more review/orchestration then writing code. That is measurable in story points over the last 8 months that we have been using AI heavily.

        The trick (and I’m not saying I have it solved but I work hard on this with some success) is making sure the AI doesn’t bloat the documentation to the point where it can’t be understood. Unambiguous diagrams and meta languages are easier to understand and validate for correctness by developers and AI alike. The more we rely on the ambiguous natural language to explain everything, the less understandable it is to anyone.

        • TheAgeOfSuperboredom@lemmy.ca
          link
          fedilink
          arrow-up
          3
          ·
          1 day ago

          Thanks! That’s really helpful. I’ll have to keep pushing back on the team to make sure they get better at their own review instead of just dumping it all on me. I still think it’s important to have an understanding of the system, and more review will help.

    • FizzyOrange@programming.dev
      link
      fedilink
      arrow-up
      2
      arrow-down
      25
      ·
      1 day ago

      The code looks fine tbh, but I know it’s an illusion.

      In my experience, it isn’t. At least as long as you use a very recent model, a language with strong checks (basically Rust), and you have a good test set, or a specification that AI can derive tests from.

      2x

      Rookie numbers. Once you accept that you aren’t going to be able to closely review all the code, and you switch to testing, 2x is laughable. AI can produce working products hundreds of times faster than humans. And yes I do mean working.

      • MagicShel@lemmy.zip
        link
        fedilink
        English
        arrow-up
        11
        arrow-down
        1
        ·
        edit-2
        1 day ago

        Curious how that works out for you. Especially when you’re troubleshooting unexpected behaviors. Because AI is pretty garbage at understanding why things go wrong. I do prod support every day and it correctly explains the problem <30% of the time.

        Also, I catch subtle things AI code does wrong at least once a week. That’s just what I catch. Imagine how much I must miss at these volumes. No, software is going to be a dumpster fire for the foreseeable future and then we will emerge from the ashes as we always do.

        • FizzyOrange@programming.dev
          link
          fedilink
          arrow-up
          3
          arrow-down
          12
          ·
          1 day ago

          Because AI is pretty garbage at understanding why things go wrong.

          Lol what? That’s probably the thing AI is best at.

          I do prod support every day and it correctly explains the problem <30% of the time.

          I suspect you’re holding it wrong. What kind of product is it, and how do you use AI? I can probably help if you want?

          Also, I catch subtle things AI code does wrong at least once a week.

          Me too, but I catch subtle things that humans do wrong more than that.

          • MagicShel@lemmy.zip
            link
            fedilink
            English
            arrow-up
            5
            ·
            edit-2
            24 hours ago

            I suspect you’re holding it wrong. What kind of product is it, and how do you use AI? I can probably help if you want?

            Well, my prod support toolkit pulls data from anywhere from 6 to 16 different systems to local jsonl files with PII obfuscated so it doesn’t get sent to the AI. Then it has to examine where data integrity was violated, assign it to the system or user that mishandled the data, discover the root cause — because if it’s an automated system then either a business rule went wrong, data was previously polluted, or an agent mis-entered data in some upstream system. Lastly, it has to identify how to fix the data and request permission to resolve the issue, or provide instructions to other operators with permissions in systems I can’t reach into to fix, such as prod database or Salesforce, which is the domain of other teams.

            It diagnoses anything from user error to system instability to networking issues to user error to business requirements that work cross-purposes. It queries three different data stores, two different log providers, and ten or so APIs.

            It has several playbooks for processing “standard” problems, which include instructions to attempt to falsify its conclusions. It also has documentation of each system and specific conventions/apis/etc. And to the degree possible, it executes everything deterministically with small python scripts for each system, “flow” bash scripts for executing standard sets of calls, and transforming and validation. It also executes multiple adversarial agents that attempt to disprove its conclusions or rewrite unclear wording and remove all the little one-time notes it leaves for itself.

            I don’t want to suggest this toolkit isn’t extraordinarily helpful, because it has saved me countless hours, but it gets conclusions wrong all the time and I have to continually ask it whether it performed this check or that one, or why it decided this was the root cause when there isn’t enough information to make that determination or there is clearly evidence that points in another direction.

            It is written pretty much exactly as I envisioned and when I need to add another tool for another system, it does that itself in minutes. There are likely some areas that could be improved.

            I recently added the PII obfuscation because someone would push back hard on that in a security review once we explain what we’re doing. It would work better if every system wasn’t written by separate teams to separate standards, but it does really well. I can compare tokens to see if they match without decoding them, and I also have a script for turning tokens back to clear text so that I can see the values without sharing them with the cloud.

            Let me know if you see any gaps there other than operational because I don’t get to dictate business requirements. I think it’s extraordinarily robust, but that doesn’t help it diagnose the right problem.

            Point is: if it has all of that and can’t reliable reason about how systems work together, what are the chances it’s misunderstanding how the code needs to go together?

            • FizzyOrange@programming.dev
              link
              fedilink
              arrow-up
              2
              arrow-down
              6
              ·
              23 hours ago

              You didn’t describe exactly what your setup is (or even what your product is, but I’m guessing some kind of CRUD web system…). Here’s what I would do if I was in your position:

              1. Set up a sandbox with Opus 5.5 and an agent harness in it (e.g. Codex, Pi or OpenCode).
              2. Most importantly, give it access to your source code, and a way to run a dev environment (set up the database etc.).
              3. Feed it all the info with PII removed as you are (good work btw).
              4. Write some skills, based on your previous experience - you mentioned having to tell it to check things repeatedly, you can just write those things down for it.

              The absolute most important things are that it has access to the source code (and documentation if you have any lol), and an environment where it can actually try things out. If you think about it, that makes things way easier for humans too so it’s not really that surprising. If I gave you a bunch of C++ and said “where’s the syntax error”, it’s a much easier task for both humans and AI if you have access to a C++ compiler.

              • MagicShel@lemmy.zip
                link
                fedilink
                English
                arrow-up
                5
                ·
                22 hours ago

                It’s in the hospitality industry. I have to make sure the separate guest profile systems align between bookings, crm, identity, and authentication. Each has a unique data store with different requirements (booking data is allowed to be bullshit because we just want that sale with zero friction) but eventually the systems must be aligned, so the system has to join identities between all systems based on matching a preponderance of PII, while also allowing someone to get a name change through marrying, move to a different location, go by a nickname, or someone made up a birthdate because they couldn’t guess. Also we have to not incorrectly merge because that creates a data spillage.

                Fun times.