Project Management AI Tool Showdown Transcript

Welcome to Future of Work 2.0. I'm your host, Ross Martin. Today, we've got a treat in store for you. It's the PM AI Tool Showdown. Our competitors today are PMI Infinity from the PMI Institute, PM Otto from Professors Antonio Nieto-Rodriguez and Ricardo Vargas, and of course, ChatGPT-4 as the incumbent. What we're going to be doing today is I'm going to be submitting five prompts to each of the tools and comparing their output. And we'll see, and then I'll rate them based on the quality that we get back. They're all going to be related to the kind of thing that a project manager might be asking an AI tool to do, so we can see how it works.

All right. So our first prompt is, what is the difference between program and project management? So I will copy it here into PMI infinity and see what comes up. Actually, a good way to do this, I think, is let's paste it into PM Otto as well and into Chat GPT at the same time. There's no reason not to go for all of them at the same time. So as we see, PMI Infinity, which is relatively new, is starting to come up with a number of some definitions of the difference between program and project management. My gut says that PMI's tool would probably be the best at this, but let's see what goes on here. So we've got a little bit here about the definition, focus, and within the text it talks about what projects are like and what programs are like. On PM Otto, Oh, this is interesting here on PM Otto, actually. Instead, what PM Otto is doing is it's not breaking it down like PMI Infinity did into all these different pieces, but giving you just an overall difference between project management and program management. And ChatGPT explains the differences and then breaks it down. So it's actually giving you a little bit of both. And so I'd ran this previously and pasted in the answers I got. And here they are. Here's PMI Infinity, here's PM Otto, and here's C hatGPT-4. So looking them over, I read them and I rated them and I gave PM Otto the 5, ChatGPT -4 a 4, and PMI Infinity I gave a 3. I felt like this one from PMI Infinity actually was, you know, They're all fine, they all explain it well, but I pretty much like the way that PM Otto kind of gave me the answer that I was looking for, which was something pretty basic. ChatGPT though, you could argue pretty easily that ChatGPT tied with PM Otto on that one, but anyway, I get to decide.

So prompt number two, I said, Create a project charter for an AI automation project. And I want that project to integrate AI chatbots into customer service function of a consumer products company. It should take three months. And I would like a detailed charter. And I want the AI to identify areas where I need to provide more than the information that I've done here so far. So let's go in and provide this to our tools. PM Otto. And I'll do the last one here, chat GPT-4. While those are working, let's go back here and see what is coming up. So PMI Infinity is creating kind of a basic project charter here, and it's got some project objectives, the scope, that sort of thing. It's asking me to figure out who my stakeholders are, some risks it's suggesting, governments, and giving me some thoughts around what's missing from here in order for me to do this right. Um, here's PM Otto. Uh, this one's interesting and I think it's because it's built on Chat GPT -4. Um, it's, it's at least formatted better and easier to, to read through. Oh, and some milestones are coming in here. Interesting. Some suggested milestones, uh, and even suggesting that I kick it off in July of 2024. Okay. And a budget. I think that's a little hard to figure out at this point. Let's see what ChatGP T-4 has been doing here. Because PM Otto is built on ChatGP T-4, I expect them to be similar. There's a little less information here in ChatGP T-4, and it again suggests where I should make some changes. So looking at what I got in my previous lookups, PMI Infinity gave me this, PM Otto gave me this, and Chat GP T-4 gave me this. These don't show up terribly well when paced into a PowerPoint slide, but that's why I went to my next question. But for now, PM Otto I still think takes the lead on this one. It's a little bit better. Chat GP T-4 is right up there. and PMI Infinity is a little bit behind.

So now I said, okay, create a Word document and give me a link to download because I realized that what I really want is I want a Word document. I don't want, I don't really want them to paste this directly into the screen. I actually want something I can then use. This happened to me last time as well. PMI Infinity said, sorry, I can't do it. And then PM Otto, though, is working on creating the document, and so is ChatGPT. So I don't want to wait for those things or make you wait for those things. So the point is, is PMI Infinity didn't give me anything. The last time I did it actually provided me a link to a project charter template they have on the PMI website, but it had nothing to do with my project that I was asking about. while PM Otto and ChatGPT ended up giving me the same well-formatted Word document, actually. It was quite nice. And so I thought, this is a great place to start a project charter. So I gave them both a five, and then PMI Infinity, I gave a one, because it didn't actually succeed in giving me what I was looking for. Interesting how I've really hung up PM Otto and ChatGPT -4 right now on giving me this Word document. So I'm going to stop those. So the last thing I want to do is get hung up on that.

All right, prompt number four. This one's a little more complicated. I'm starting to think, okay, how do I push it a little bit here? Create a list of the potential risks for a PPM SaaS tool implementation. Rate their likelihood and severity on a scale of 1 to 5, 5 being highest, and suggest 2 to 3 mitigations for each risk. Provide that in a spreadsheet, make sure each mitigation is in its own row, and make it an Excel download. So that's an interesting way to go. That's where you're really being specific, saying, I want AI to figure this stuff out for me, and at the end, create it as an Excel download. And I think that's a good way to go, because on these types of things, if you don't ask for that, it'll just show up on the screen, which isn't necessarily how you need it. So let's go back to PMI Infinity. And this happened to me last time. Um, and I think this must be about PMI infinity. Cause they couldn't do the word document either. They, I asked for it as an Excel download and it could not give me an Excel download. Now, the risks and mitigations that it's showing here are actually pretty good. It didn't follow my rule, though, where I wanted each of the mitigations to be in its own row. And what's interesting here is it says you can download the spreadsheet here, but this isn't actually a link, so there's nothing I can do about that. PM auto is still running chat cheap TP finished and here it is. I've got my file and Let's see what we got and Opening up Excel Make it a little bit bigger for all the rest all of you guys. Sorry about that. I So yeah, so this is kind of an interesting thing. I probably would want to have it group it a little bit better It did do what I asked and Interestingly enough Yeah, and it's not sorted the risks aren't sorted correctly. So it's hard for me to completely tell like let's go to data auto filter We'll sort by risks. And now you can see that actually, I've got three mitigations for data security and privacy concerns. You know, three, three on all of them, actually. So that worked out pretty well. And then let's go see what's going on with PM Otto. Here comes PM Otto's contender. Let's open that one up. I have a feeling it's probably not going to look massively different than the one from ChatGPT. But, well, actually it does. It's much smaller. It's sorted properly, which is nice. But now there's only two per... Also, they only have five issues. And if I look at the one from GPT, they've got six risks, I mean, not issues, risks, and three mitigations each. So a little bit more complete answer. And that's what I found before when I was doing this. So let's go look. PMI Infinity, I went ahead and put their stuff into the form I wanted. It was pretty good, but I didn't like the form. PM Otto was a little light. And ChatGPT-4, I thought, did a nice job on this. So in this case, I actually gave it to PMI Infinity. And the reason is, yes, they didn't follow my rules properly, but they covered what I was looking for. And so I felt like PMI Infinity did a nice job of giving these things. Once I personally broke them into the form I wanted, which annoyed me, but I decided to go on content over form.

Alright, prompt number five. Create a high-level project plan for an SAP S/4 HANA implementation to replace multiple existing legacy platforms across five countries in Western and Eastern Europe, and provide it as an Excel download. So that's interesting because, and I'll start with ChatGPT and PM Otto this time on these because last time they took so long. That one's an interesting one because it's actually a project that I, a real project that I was involved with at a previous company. So It'll be interesting to see what these guys can come up with. It's very technical. You need to understand SAP, you need to understand what kind of a project plan you need to do. Now, as you see, PMI Infinity is coming up with some stuff right now, but not as an Excel download again, which is the same problem that we had before. It's missing the ability to specify how you want to receive the information. Let's see what's going on with PM Otto. Oh, they're still thinking. And ChatGPT, ah, ChatGPT has actually got it. So let's see what we come up with here. All right, come on. There we go. All right, let's see what this project plan looks like for ChatGPT. Oh wow, this is really weak. There's nothing here. Okay, for Chat GPT for SAP project plan. Now, in its, you know, in defense of all these AI guys, I didn't give them much. It's really, you really can't start with a project plan, you need to start with the project charter. And before you start with the project charter, you need to figure out what the objectives, the strategic objective you're trying to get figured out before you can do a plan. But still, It does mention SAP S/4 HANA on here a couple times it mentions countries But basically I just is not good at all So, let's see what's going on here with PM Otto. Well, it's still working on it and for and then for PMI infinity It looks like a real just a generic Project plan like at the highest highest level I really not much I can use other than to maybe prompt me and thinking about how to create a project plan like this Let's see what's going on with PM Otto. Ah, here we go All right, PM Otto. What'd you come up with? PM Otto. Whoops. That's the other one Let's go here. Come on. You can do it Well, okay, hmm, interesting. Still extremely high level and vague. They got some start dates and some durations and some things here, but even some assignments, who's supposed to be doing these things? Still, I would say none of these things are things you can actually work off of specifically. And that's what I found when I did it earlier before the video. For PMI Infinity, I found that it just kind of explained how to create a project plan and then gave me some basic stuff here. PM auto, like that this is what we just saw here. Some real basic information, not too much to go by, but interesting. And then what I found from ChatGPT-4 last time, it's interesting how ChatGPT-4's version this time was different actually, in some ways worse. But very little here, just very little to work with. So what I said is they all kind of failed this test. But again, in its defense, AI is not that sophisticated and it doesn't know enough about SAP implementations in Europe or anywhere else to really do this right. So I wanted to see if I could really push it to the edge. And so I gave them some lower scores.

Anyway, so here's our final results. Basically, it's almost a tie between PM Otto and C hatGPT-4, but I'm giving PM Otto the edge just because it did a little bit better job on prompts 1 and 2. PMI Infinity still has quite a way to go. And so that's our winner. Our winner is PM Otto. Now, keep in mind to use PM Otto, you have to have ChatGPT -4. PM Otto is a custom GPT that lives on top of ChatGPT -4. And ChatGPT -4, if you don't know, requires you to have the plus subscription to ChatGPT. So it's not the free one, it's the $20 a month one. But I find that it's a worthwhile $20 a month for Cha tGP T-4, and now I have PM Otto on top of it, which is great. So the general thinking at the end here about this is, let's go back up to the top and talk about the showdown. So what do we find here? Well, I'm really pleased that there are starting to be some AI tools specifically thinking about us project managers and how we do things. But they're pretty early on, pretty nascent, right? They're just starting to get And I know that Antonio and Ricardo and the PM Otto guys are continuing to pump more and more information into their large learning model for PM Otto to try and get it to be more understanding about projects and thinking about things like risks and learning how to make project plans. We are early days on PM AI guys, really early. The fact is, is that we are, you really are only able to do some basic things. You're able to take meetings and have transcripts made from them that then pull out the key lessons learned or the action items from the meeting, and you can send those things out. You can do some basic things. You could pump in your project charter into ChatGPT or PM Otto and ask it to identify the risks, and that will help, but you're still going to have to add more onto it. So nothing that PM AI tools are doing right now replaces a project manager. At this point, It's not even really your best project coordinator assistant that you might have working for you. But it is an assistant, and I have found it to be useful in at least getting the ball rolling, so I'm not working off of a blank sheet of paper. When I'm working on a project, I can get it to help me think through my initial plan, or help me think through my initial communication strategy, or change management strategy, or risk assessment. all those plans that you used to have to do all by yourself, it can give you V1, the draft V1 of those. You have to do the rest, it can't do it all, but what I do know is every time I've submitted a project where I gave more information into the AI tool about my project than these little prompts that I just did. That's when it made a big difference. It understood what I was trying to say and its answers were much better when I gave it more. Anyway, thank you very much for joining Future of Work 2.0. I hope this has been helpful, and I appreciate your viewing and listening to this. If you like, please subscribe, join our newsletter at futureofwork20.com, or like us or anything else. We really appreciate it. Thanks a lot.