[Title slide 1. Blue CAI company logo with tagline “We power the possible” appears in middle of screen. Company website www.cai.io appears at the bottom center of the screen]
[Title slide 2. Solid background. In the middle center of the slide, appears the heading Webinar On-Demand. Underneath of the heading appears the title “Generative AI and large language models in IT service and support”. In the bottom center of the slide, the blue CAI company logo with the tagline “We power the possible” appears]
[Presentation slide 1. Black background with image with four speakers appearing on the top left of screen. White box with the presentation title “Generative AI and Large Language Models in IT service and support”. Line below title reads “Live Date: April 11, 2023”. Line below date reads “Sponsored by”. On bottom left, blue CAI company logo appears. On bottom right, multi-color HDI company logo appears.]
00:00:00 - 00:00:27
Tim McElgunn
We are very glad you've joined us for Generative AI and Large Language Models in IT Service and Support. I guarantee you we're going to go far afield from that title over the course of this conversation, but we will do our best to pull it back to that topic. I'm Tim McElgunn. I'm the editorial manager and principal analyst at HTI and I'll be your host for today's session.
[Presentation slide 2. Black background with image with four speakers appearing on the top left of screen. White box with the text “ON24 Console Logistics, Optimize your experience today.” Five bullets appears below this text, which include “Streaming or Audio Issues? Simply hit your F5 key (PC) or Command Symbol + R (Mac) to refresh your webcast window.”; “Enable pop-ups within your browser”; “Turn on your system's sound to hear the streaming presentation”; “Questions? Submit hem to the presenters at anytime in the Q&A window”; “Technical problems? Click "help" or submit a question for assistance”]
00:00:27 - 00:01:04
Tim
Just a few housekeeping announcements before we get going. If you have any technical difficulties during the session, simply hit F5 or command R, depending on your platform, to refresh your browser browser window. If you need assistance beyond that, just click on the yellow help icon below the slides and somebody will get with you right away. We do encourage you to ask questions at any time during the discussion, do that. Simply type your questions into the Q&A window on the side of the screen and hit the submit button. Due to time constraints, we will probably only be able to answer a limited number of questions today, but we will try to get back to you with responses if you request those from me directly as soon as possible after the session.
00:01:05 - 00:01:29
Tim
There are some additional resources related to this discussion that can be found in the resource list widget located in your console. I encourage you all to take a look and download those. Then, finally, today's presentation will be available on demand shortly after the completion of this live event. You'll receive an email reminder once that becomes available. Thank you very much to CAI for sponsoring today's discussion.
[Speaker slide 1. Black background with image with four speakers appearing on the top left of screen. White box with the text “Featured Presenters. Our knowledgeable speakers today are:” followed by four images of white men wearing business attire. Under the images appear their names and titles: Matt Peters Chief Technology Officer, CAI; Marc Seybold Nemertes Research Fellow; Michael Azoff PhD Chief Analyst, Omdia; Tim McElgunn Principal Analyst, ICMI and HDI. On bottom right of screen appears multi-color HDI company logo.]
00:01:30 - 00:02:12
Tim
With that, I am pleased to introduce today's panel starting with our sponsor representative CAI's chief technology officer, Matt Peters. [unintelligible 00:01:39] at Accenture in Oracle. Matt brought his leadership skills and focus on applying innovative technology to client challenges to CAI. In addition to running enterprise technology infrastructure and security operations, in his spare time, Matt also heads up all of CAI's technical consulting practices. Matt's a frequent speaker at national conferences, webinars, and podcasts. He is a music aficionado attending, over 300 concerts a year, I guess in his life. He enjoys very much hiking with his family in national parks across the country. I'm very glad you could join us today, Matt.
00:02:13 - 00:02:15
Matt Peters
Thank you for having me. I appreciate it.
00:02:16 - 00:02:49
Tim
Michael Azoff is the chief analyst on Omdia's cloud and data center team where he covers a range of topics related to the cloud, data center, AI, software development, agile and DevOps, consults directly with clients and supports in former tech events. Michael's career includes senior consulting and analysis roles at GigaOm and Kisaco Research. I assume I pronounced that wrong, and in Omdia's predecessor company Ovum Research. Michael's also worked in high-tech R&D, built neural networks in a startup and published a book Neural Network Time Series Forecasting of Financial Markets. It's great to have you Michael.
00:02:50 - 00:02:53
Michael Azoff
Thanks. Good to be here. Thank you.
00:02:53 - 00:03:25
Tim
Finally, we're very pleased to have Marc Seybold join the conversation. Marc is a research fellow at Nermetes ... Help me with that, Marc. Pronounce that for me, would you? Okay. It's one of those things, I've been reading it for years and I've never pronounced it out loud. Nermetes Research focused on emerging technologies. Marc's a native New Yorker who received his BA in computer science from Queens College City University of New York, and he is the former CIO of SUNY College at Old Westbury, New York and chair of the Council of Chief Information Officers SUNY. Welcome, Marc.
00:03:25 - 00:03:27
Marc Seybold
Thank you very much. Glad to be here.
[Polling slide 1. Black background with image with four speakers appearing on the top left of screen. White box with the text “From what you know about generative AI and large language models and its potential use in IT support services, are you overall”: Below the polling question are four options: A Positive; B Negative; C Cautious and awaiting developments; D Undecided. Below the polling options appears the word “Submit”.]
00:03:28 - 00:04:01
Tim
Okay, let's see. I'm going to start with a quick level set. We're going to run a poll here. Probably leave it up for about 30 minutes. 30 minutes, 30 seconds. If you all would take a second and go ahead and answer that. We'd just like to kind of get a feel for where you see things as of this moment. Okay.
[Polling slide 2. Black background with image with four speakers appearing on the top left of screen. White box with the text “From what you know about generative AI and large language models and its potential use in IT support services, are you overall”. Results displayed in a bar format as follows: Positive 50%; Cautious and awaiting developments 40%; Undecided 10%; Negative 0%]
00:04:02 - 00:04:42
Tim
Okay. Janine, how are we looking on the poll results? Can you pull those up for us? What do we got here? Everybody's positive. Wow. Not everybody is positive. A good number of people are positive. Let's see. I'm trying to pull up that slide so I can see it properly. I don't know why it's not doing what I'm asking it to do. Let's see here. Okay. Janine, if you could help me get this thing up on my screen, I would appreciate it. What's going on here. Janine, can you see the poll results?
00:04:43 - 00:04:44
Michael
I can see it. Do you want me to quote it for you?
00:04:45 - 00:04:48
Tim
Yeah, if you would. I don't know what [unintelligible 00:04:48].
00:04:49 - 00:04:59
Michael
Sure, no worries. 59.1% positive, zero, negative. 31.8% cautious and awaiting developments and 9.1% undecided.
00:05:00 - 00:05:27
Tim
Okay. Well, I guess it is to be expected. We've got a bunch of technology folks on the line. The fact that nobody's negative on this, cautiously optimistic, I guess that's to be expected, but I am a little bit surprised that nobody's sort of feeling a little bit more cautious than that. Okay, great. Well, thank you all very much for that. Let's get on with the show here.
[Speaker slide 2. Black background with image with four speakers appearing on the top left of screen. White box with the text “Questions? Submit questions to the presenters via the Q&A window” followed by four images of white men wearing business attire. Under the images appear their names and titles: Matt Peters Chief Technology Officer, CAI; Marc Seybold Nemertes Research Fellow; Michael Azoff PhD Chief Analyst, Omdia; Tim McElgunn Principal Analyst, ICMI and HDI. On bottom right of screen appears multi-color HDI company logo.]
00:05:28 - 00:06:15
Tim
Recognizing that this space is evolving as I have daily, it's hourly, it's by the minute, at [inaudible 00:05:35] point. The space is evolving so quickly. Let's take a look at where things stand in terms of large language models being applied specifically to support IT's service strategy. Let's start if we could by differentiating for our audience between public or open models, Chat GPT being the most famous at this point and perhaps more proprietary large language models,, if there are any that are being developed specifically for IT support solutions. I'm sure we'll get into the ones that are being developed for other applications as well. Starting with you Matt, and then Michael, and then finally Marc. Can each of you share your perspective on how these models can differ and where they're being applied right now?
00:06:16 - 00:07:13
Matt
Yeah, I'm happy to. Actually, I'd say I appreciate both, you identifying this difference, and pulling it out at the front of the conversation, because in a lot of my conversations with a lot of clients, this is a pretty big point of confusion tied to LLMs right now. I think you fall into two camps for most people. Either, folks that don't realize that there is a meaningful difference between a very large, very open model trained on an extremely huge data set versus very purpose-driven and focused models. Then there's a whole other group that realize that there's a difference, but they misunderstand the limitations between the two. There's a general feeling that, well, if you're a wide open model, a big model if you will, then you have access to the world, you're going to get more right, you're going to have more capability, you're going to be a more powerful tool. If you are a focused and smaller model, then you're not going to be able to do as much with that kind of a tool.
00:07:14 - 00:08:02
Matt
In the context, especially of IT service delivery, but a lot of use cases, that's not necessarily true. These two types of models have an important relationship to one another. The very large, open models are really critical, because probably we're going to continue to see that most of the innovation, most of the new discovery is going to happen there, but that does not equate to them being right more often, or more accurate more of the time. None of that. The smaller and more focused, purpose-driven models that are designed to address a really specific use case, IT service support as an example, they're perfectly powerful on their own. The big distinction between the two is largely in what they have access to, in the context of responding to a user, answering a question.
00:08:03 - 00:08:38
Matt
When you think of about something like Chat GPT, it has access to the whole world. If I ask at an IT support question, sometimes I'm going to get a perfectly good answer, sometimes I'm going to get a completely ridiculous answer skewed by social media and a wide amount of misinterpretation from human users that the system can't discern the difference between. If I'm looking at a very purpose-driven model that's meant to do IT services within my organization, and several of them exist, I'm much more likely actually to get a correctly curated, accurate response, something that's actionable that I can actually do.
00:08:39 - 00:09:27
Matt
Or, on the flip side, I have a much higher probability that that solution is going to come back at me and say, "I'm not qualified to answer your question," or, "I don't know the answer to your question. You need to do this instead." Unlike, what tends to get the bigger models in the news all the time, which is hallucinations and completely incorrect responses given with all the authority and confidence in the world, which when you think about the user community, in the context of IT services that we have to take care of, that's dangerous. That's a thing that's worth worrying about. It's an important distinction between the open models and the purpose-built models. They have a meaningful role to play, both of them, but that distinction is very important for how much you can trust it, how much you're putting at risk, if you will, in your company, if you choose to deploy one.
00:09:28 - 00:09:43
Tim
That's interesting, because the other side of that with the small focused model is that it will reveal, for example, gaps in your knowledge base very quickly, which will be extremely useful I think for IT services and support. Michael?
00:09:43 - 00:10:43
Michael
Yes. The open solutions, at the moment, are freely available, some of them. This is, I think, not going to last, because it's very expensive to run these systems. We can already see Open AI is monetizing its technology through partnerships and OEM opportunities. There's also a risk involved in putting these out freely available out in the open in that if it goes wrong, it can really backfire. Clearly there's been a step change. We've seen a number of ventures, from some of the other technology companies, where they've put out systems and it's backfired on them and they hastily had to retrieve them, but clearly something has changed. Open AI system is a step change in improvement.
00:10:44 - 00:12:10
Michael
I think, in terms of the proprietary systems, I think the nearest comparison, I would say, would be the intelligent virtual assistant market, which is heavily based on AI. I can see them also taking a huge interest in bringing in LLM generative AI type of technology into their solutions. Also, the IVA solutions are not just text based, they're also voice based. I think what we can see is how those systems, unlike Open AI, which says our solution makes mistakes and is not mission-critical ready, IVA solutions have a much greater accuracy. They're based on an individual company's needs, so they can answer questions particular to that organization. They also integrate with backend systems. All of those things are important. I think where we're going to see solutions like Chat GPTI is how they're embedded in these other solutions so that they can actually make better use and provide a superior interface.I think there's going to be a lot of interplay between these open systems and the proprietary ones.
00:12:11 - 00:13:29
Michael
Yeah. Also, just to look at [inaudible 00:12:19] the open players, I was looking at the Allen Institute's Macau and it had a logical reasoning example, which it put before its system and Chat GPTI, which was basically a logical reasoning ... The suspect killed the judge, who did the police arrest? The Macau system said the suspect and the Chat GPTI said the suspect's brother. Now, clearly that's quite a comical response. I did wonder whether that was an old Chat GPT that they ran it through. I ran it on the most current one this afternoon. The first response was very if and but and full of regulations and not really committed. I then rephrased the question, I said, "This is a hypothetical exercise in logical reasoning," and then put the question, the suspect killed the judge, who did the police arrest. It came up with a better answer.
00:13:30 - 00:13:37
Michael
Yeah. There's some interesting things there. If we have time, I'll give you the output from Google's Bard, but I better stop now.
00:13:38 - 00:13:54
Tim
Great. Yeah. I guess that first answer makes me think that vaudeville is also part of the input for these models. I have to watch out for not only jokes, but bad jokes going forward. Marc, what's your view on sort of differentiating between these systems?
00:13:55 - 00:14:49
Marc
What I'd add to what's already been said is that there's a tension between having the smaller models that can be kept closed and proprietary, and we don't have to worry about data leakage and the desirability, especially if it's some kind of support system in IT, of being able to do one shot, low shot learning that's peculiar to your organization or your environment. The hope is that there are some models, we talked a little bit about it before the seminar started, what Stanford did with the metal model. Alpaca Lama I think is the name they're calling it, where it's possible that you might be able to get the models big enough to go through that phase change where they are capable of still learning, but small enough that they don't run into the problems of the much larger models where you're getting the hallucinations and the garbage in, garbage out type situation.
00:14:50 - 00:15:06
Marc
Of course, they're far cheaper to run. The Stanford one was $600 to train. It will run with an acceptable number of token outputs on desktops and laptops. That's the kind of thing that you can see being embedded directly into systems themselves, even locally.
00:15:07 - 00:15:25
Tim
It's going to be interesting to see some of these decks that the VCs are going to be seeing from the various players talking about the cost to train and the quality output and trying to make that decision and figure out how to balance those factors. Marc, we're going to stay with you.
00:15:26 - 00:15:27
Tim
I'm sorry. Go ahead. Yeah.
00:15:28 - 00:16:07
Marc
All right. Just to finish that thought, the other thing that's kind of a wild card here is there's a large number of startups that build neuromorphic chips that are aimed at the IOT market. All of a sudden the fact that you can get these models that are small enough to run on a laptop, a desktop, opens up an enormous market for all of these players, which means that it might be possible to get something that's totally standalone with capabilities that approach the larger ones without their downsides, either from a computational cost, price of running, execution, purchase. It's a wide open field there. There's a lot to unfold yet.
00:16:08 - 00:16:27
Tim
Yeah. That's really interesting, in the IOT context especially. I mean, not only are we talking about a bunch of network things that are running off the same model, but potentially individual devices that are carrying their own models on board. A couple more things to worry about.
00:16:28 - 00:16:39
Marc
[unintelligible 00:16:28]. Sorry to step on you. The other interesting question is what happens when there are extremely large numbers of these things interacting?
00:16:40 - 00:16:43
Tim
Right, and learning from each other.
00:16:44 - 00:17:01
Marc
Exactly. They're all communicating, they're all kind of learning from each other, exchanging data. Do you get an emergent property that's significantly different than the emergent properties that we've seen in the large language models that are running on centralized systems? The Bards, the Chat GPTs?
00:17:02 - 00:17:03
Michael
Yeah.
00:17:03 - 00:17:05
Tim
That's fascinating.
00:17:06 - 00:18:11
Michael
Just to make bring to your attention that there was a very interesting paper that came out of the Google Research Team on emergent abilities in very large models. They found that once you had a critical number of parameters, so that means essentially connections between neurons, once you get to a certain size, 170 plus billion, then you start seeing that the LLM model is able to come up with some solutions for specialist benchmarks that have been designed to make it difficult for AI to answer those benchmark questions. They were going from zero responses, suddenly, once you hit that sort of high billions of connections, you were getting 40% accuracy. There's a step change and something interesting is happening when you have these very, very big models. That's something, I think, there's going to be some interesting research on that and implications. We're going to see so much more, I'm sure.
00:18:12 - 00:18:36
Tim
Yeah. It's hard to realize, just because of the sheer volume of discussion around this stuff already, but we're just on the doorstep. We're nowhere near where we're going to be, even by the end of probably this month and certainly by the end of the year. Yeah. So much to talk about, thank goodness, because that gives us all reason to be here this afternoon.
00:18:37 - 00:18:53
Tim
Marc, getting back to sort of the difference between these large open models and perhaps more proprietary focused models, are you seeing anything or are you seeing some stuff? What are you seeing in terms of these less public or more proprietary models?
00:18:54 - 00:19:49
Marc
I can't talk about the client projects, but what I can do is talk about the questions that we're asked and that gives a hint as to where other things are going. LLMs and cybersecurity, everybody wants to know what does this mean for incident response, on an automated basis, as well as live alerts of an attack. People are asking about knowledge capture. This goes a little bit back to the one shot learning, but it's far beyond that. All of us have information in our heads that never gets written down about how we do our jobs and, more importantly, maybe how we actually get things done inside an organization. Is there any hope that this technology would allow that information to be organically captured so that as people come and go, that's still retained in the organization and can be easily queried to maintain continuity and to improve services?
00:19:50 - 00:20:50
Marc
Surprisingly, we're asked, even for basic things like we had one C-suitor come in and say, "If this thing could summarize a one-hour Zoom meeting that I wasn't in, that alone would be worth it to me as an advance." Cobots, in the sense of robotic process automation that's right now mostly focused on back office work, how far does the needle move on that when you add LLMs? How much further up in somebody's work chain can you go? Then there were the really kind of blue sky ones where if it was theoretically possible to livestream all of your corporate email, all the Slack channels, your data tracking, task management, everything into that system, how far away is the day when you could do something like have HR push a button and performance evaluations are generated for everybody in the company and then supervisors could simply, either sign off on them or make modifications.
00:20:51 - 00:21:37
Marc
Having all of that information, how far away is it where you could make a general statement saying, do this, create this marketing program for a new product that's coming out, it go ahead and implement this particular IT technology, and then the system would actually break that down into tasks, survey who's available to do it on the basis of their skill sets, but also which assembly team members would create the least amount of friction, least amount of conflict in accomplishing those goals and then monitor their progress and kind of nudge them whenever they would get stuck. People are thinking everything from nuts and bolts to almost the sci-fi type things.
00:21:38 - 00:22:33
Tim
Pretty interesting. Yeah. On the other side of my job is to look at the contact center industry, and of course that they have a lot of very similar issues to IT support. One of the things that early AI applications that we've seen is this sort of live coaching where the AI is listening to the call and is able to step in and help, presumably, a relatively junior contact center agent respond very, very effectively. That's going to be interesting, because one of the issues that I've been thinking about is, as this automation comes in, whether it's on contact center or IT support, where do the newbies learn? If the machines are taking on a lot of the basics and the grunt work, how do you become a level two or level three tech? Where do you get started? I think that that's going to be a very interesting application for this stuff and around knowledge management as well.
00:22:34 - 00:23:12
Tim
One of the things you said that caught my attention, it's a different context, but you talked about incident response. I'm thinking in terms of the ITSM, IT service management mode, where there's incident and there's problem management and there's all this relationship management, all these other pieces that go into the model. It's already structured. It seems pretty interesting if you take that from an IT perspective and start applying some of these models to something like ITSM or covid, any of the other frameworks that are out there. Again, that's a rat hole we probably don't want to go down too far.
00:23:13 - 00:23:27
Tim
Matt, what have you looked at so far? What are some of the examples you've looked at or maybe some of the questions you're getting from clients that are indicating where people are looking in terms of IT service and support with these models?
00:23:28 - 00:24:14
Matt
Yeah. I mean this is tied very tightly to some of the services that CAI provides to customers. We're looking at this all the time. I'd say, at the moment, we're seeing a lot of curiosity around open models and what they might be able to be leveraged for. I think our client base skews a little bit more toward the cautious side, than the poll at the onset of this session suggests. They're a little bit worried about what are they giving away, what risks are they creating, what kind of data leakage threats are a potential there. That pushes them a little bit more to proprietary models right now in the context of their investigation, but we're seeing a lot of activity there as well.
00:24:15 - 00:25:16
Matt
I think that's an area where the evolution is quick, maybe not quite as quick, but they've come along a very long way. A lot of the play there seems to be in the intelligent virtual assistant space as was previously talked about. That's where a lot of that stuff is more readily manifested and can drive some pretty measurable business value for an organization pretty quickly. In the context of how do we spend, how do we make these decisions, they lend themselves a little bit more nicely to, all right, if I've got ... In the ITSM world, I mean you said you didn't want to go down that rabbit hole, but I'll go down it. In that particular world, we have a lot of data already. We know a lot about resolution rates, total contact volume for an organization. Where and when do things actually get resolved? What are our escalation rates? We know what we need to know to build the framework around. For an organization that is evaluating, deploying that kind of technology, what's the impact likely to be?
00:25:17 - 00:26:08
Matt
It's not the same in every organization. Two nearly identical organizations, in two completely different verticals, will also have a totally different outcome. There's an element here where every piece of this is very context specific and very use case specific, which is a struggle for a lot of our clients right now really trying to understand, could this be for me or could it not? In a lot of context, really just trying to get their head around what are the risks behind that. I even talk to IT leaders that don't appropriately appreciate, in my opinion, today, that if you give content to Chat GPT, that becomes Open AI's to do with as they see fit, to a certain degree. They're introducing new versions of models that put some more limitation around that, but there's a big and open risk. That's been a lot of what's been getting it in the news recently.
00:26:09 - 00:26:59
Matt
A lot of those threats don't exist in the more purpose-built, privately deployed models, but we see a lot of the same kind of functionality. A large language model doesn't have to mean that the solution is big, if you will. That methodology and that technology is applicable at every scale. We see even in our own world, in-house at CAI, we have an LLM deployed that is tied to the service desk and is designed to let people chat with it, interact, get questions about things going on in the company calendar, I need a new laptop replacement, what have you. It takes on a lot of that work. It does it very conversantly and very successfully, but it doesn't take away the need for us to have the service desk either. That's an expectation that we try to steer our clients away from.
00:27:00 - 00:28:02
Matt
If you think that this thing is fully independent and is doing that blue sky kind of work that we were just talking about, they're not quite there yet. Co-piloting, yes, very much so. Augmentation of the work that the service desk can do, absolutely. There's still a role for that individual, and a role in which they can learn, to your prior point, Tim. I see that that's a pretty big component of all of this. What I think we're going to start to see, what I hope we're going to start to see, as a product of more introduction of LLMs in that specific kind of scenario is that all of us will be getting a little bit smarter about how we measure the success of that service desk. That it becomes more and more experience driven and that we're measuring the success of the overall effort in the context of how many resolutions did we get, how positive was that experience, how much wider were we able to make the waterfront of what the desk could do for the company, as opposed to just thinking about it in terms of headcount and a cost per ticket.
00:28:03 - 00:28:37
Matt
Those are smart ways to think about cost justification right now, but if I think a little bit shorter term, but further forward for companies, that's a nice way to elevate the game for IT across the organization. As we see some generational shifts starting to happen, with the overall workforce, it seems to me to be a better vehicle to meet people where they're at and engage them in the wider variety of ways that they want to be engaged. It's been true for a while. Not everything's a phone call anymore. This is going to further drive that down, but phone calls are not going to go away.
00:28:38 - 00:29:07
Matt
If we look across our own clients, we see that really reliably. Lots of people are more interested in chat now, even if it's just live agent chat. That's a medium that they're very comfortable with, but when there's a real fire alarm emergency, it's almost always the phone. That's just not going to go away. I need a person, I need them now. I don't think that we're going to eliminate that completely. It's a while before they're going to be so good that they're fully independent. I don't even know how to project when that might come in the future.
00:29:08 - 00:29:24
Tim
Right. There's times where you absolutely want the person on the other end of the line to understand your urgency. Then that's very much a human interaction type of thing. Michael, you have anything else to add on how working with these different types of models might differ from each other or do you think we've covered that.
00:29:25 - 00:29:25
Michael
Yeah.
00:29:25 - 00:29:28
Tim
Okay.
00:29:29 - 00:30:48
Michael
Sure. We've mentioned IVAs, intelligent virtual assistants. It's a field that I did side by side vendor product comparisons and they all used AI. It was just incredibly interesting to see all the different models. This was a generation before LLM, so we're talking about some years ago. These systems are really fascinating. One of the results that I pulled out from that study was on ROI. Once the system is trained on the particular business' requirements, from day one, the IVA was being used as a first line of support in the call center and was able to deal with easy requests, something like 80% of the calls with a closure rate of 80 to 90 plus percent. That's pretty good from the first introduction. Subsequently, they were able to develop these systems further so that they can deal with second line of support. Of course, when a system like this cannot answer a query it's then able to select the right agent to pass the query onto.
00:30:49 - 00:31:39
Michael
This is a mature and successful market. I think it's going to be ... This is, as I say, is voice and chat. I think that LLMs is going to have a nice disruptive effect on this market and improve it one step further. I'm eagerly waiting to see how the IVA vendors react to LLM. I think it's going to be a watch this space, but as everything that Matt was saying, it really is about having a system that is very much overseen, governed. There isn't the kind of risks that we are seeing with OpenAI Chat GPT at the moment. That is a major, major difference between these proprietary systems, as Matt said, and the open systems.
00:31:40 - 00:32:22
Tim
Great, thank you. Keeping in mind that the audience for HDI generally, and certainly today, these are practitioners for the most part. Yeah. They're fascinated in all of our discussion about the long term, big implications of this, but they also want to understand what does it mean for their teams? How is it going to change the way people work? I'm curious, Michael, I'm going to stay with you on this one to start, what are some examples where applying large language models, whether open or proprietary, might help improve an IT team's ability to deliver top quality work and to really improve the end user experience? Because at the end of the day, that's really what we're trying to accomplish.
00:32:23 - 00:33:24
Michael
Sure. I recently looked at, and this is a paper that is available for the audience to download. This is looking at the application of these AI systems to assisting developers with coding. I counted something like 16 solutions in the market. They're not all based on LLM. LLM and [inaudible 00:32:47] is open ... The knowledge of how to use these systems is out there and there's a lot of startups making use of this technology. Assisting coding is a really good application of this technology. It is not a hundred percent. It does make mistakes. It's not going to put developers out of work. Okay? Especially if you're using it directly, you need a professional developer to assess the response and make sure there's no mistakes in it.
00:33:25 - 00:34:35
Michael
There's a lot of no-code, low-code systems out there. I think it's dangerous to have them directly interfaced with an LLM solution. You need something in the middle: guardrails, governance, to ensure that if they have access to a corporate database, they don't destroy it. Depending on who the user is and their knowledge of being able to assess the response, that will dictate how best to use it. I definitely am seeing, certainly in the coding assistance market, I'm seeing examples already out there. Also for the IT services market, I'm seeing that ServiceNow, they have something called Big Code I think it's called, which is a big research effort with partners looking at LLM. BMC, for example, is looking into this technology. I think we are going to see this technology coming out of these players in the services market for sure.
00:34:36 - 00:35:01
Tim
Yeah. Great. I had a thought, one of my [inaudible 00:34:42]. It went away. If I just had an AI assistant to follow me around like Boswell followed Johnson, I'd be in much, much better shape. Marc, what's Nermetes' perspective on this? Are your clients, specifically in the sort of IT service world, are they moving forward quickly with LLMs or is it really just sort of a look and see period right now?
00:35:02 - 00:35:56
Marc
No, it's the exact opposite. The large clients, for the most part, are actually banning it as we had that earlier discussion before we went live. They're in the reactionary mode. They're forming teams or have already formed teams and they're essentially doing a SWOT analysis. They're very nervous about data leakage, compliance issues, not having corporate policies in place. What's the ultimate organizational impact? It's naive to think that these systems can be introduced across the entire company, especially if you're talking 50 or a hundred thousand or more people, and that everything just keeps going the way it was before. Their perspective is not just the technology and the concerns there, but it's also the organizational side of it and the fact that you've got such a polarized public discussion on it.
00:35:57 - 00:36:51
Marc
On the one hand, you've got a thousand people that have signed a letter starting with Musk saying, do a pause on this, basically implying to people that Sky Net's around the corner. Then on the other side you've got the leading researchers and scientists in the field saying, "No. That's nowhere near around the corner. We wouldn't even propose a timeframe for when that might happen. We don't think it's a good idea to do a pause." In an environment like that, you're going to go risk averse and say, "We'll go with the low hanging fruit," which is if vendors are starting to integrate these things into their products, then that's kind of like a safety mechanism, it's the guardrail. For the large projects it would be initiated in house, they want a much deeper understanding of where the risks are in this and what the larger implications are.
00:36:52 - 00:37:24
Tim
Yeah. I think, to your point about this is about organizational structure, process, people, strategy. I think that the biggest challenge for managers over the next however many years is going to be that you're now not just managing people and technology, you're managing these blended teams. It is going to be an incredible management challenge going forward, for all the reasons that you touched on.
00:37:25 - 00:37:41
Tim
Excuse me. Same question for you Matt. You've talked about this a little bit already, but are they seeing an active interest in clients coming to start exploring solutions and challenges around LLMs for their specific needs?
00:37:42 - 00:38:10
Matt
Yeah. We absolutely have. I'd say it's important in that context to put those folks into two different camps. There are the people in the last quarter or so who are really worried about how do I use large language models all over my organization? What do I do? Should I be afraid of them? Should I not? It's really as much a fear of missing out motivation as anything else, but that's really in the context of just Chat GPT and the fact that you can't get away from this in the news. I understand that. But that's one piece of motivation.
00:38:11 - 00:39:09
Matt
There's another side to it, for at least a year now, that we've been talking to our customers, either us going to them or them coming to us and saying, look, now that large language models are being deployed in purpose-driven scenarios, this isn't the chatbot experience that you had three or four years ago when you tried to roll one out and it was a technology that wasn't quite ready yet. Universally, everyone was frustrated with it. You had a bad experience and we get it, but let us show you something new, something that behaves a lot more conversantly, understands what you're going for now. That's really been opening the door to a lot more productive conversations with customers around how might we be able to leverage this kind of technology in-house. Well, those more contained scenarios are certainly the easier way to get started, in the context of I'm worried about risk that I'm taking or I have contractual obligations to protect data that I have no option about.
00:39:10 - 00:39:33
Matt
I'm a victim, if you will, of that as well. CAI is governed by the agreements that we have with all of our customers. We have to worry about that too. Those purpose driven LLMs have been emerging and really starting to build some of their own momentum in this space, without Chat GPTs help, but man, did that ever turn up the volume in the last couple of months.
00:39:34 - 00:04:13
Tim
Yeah, I can imagine. We've talked around a lot of this already, but I'd like to dig in a little bit on the possible downsides that you're all concerned about. I'm not only talking about security, technology, et cetera, are also people. Let's talk a little bit about how this is really going to impact the overall practice of IT service delivery. Matt, can you expand a little bit on some potentially less positive outcomes that IT leaders, I guess, need to be ready for? They're going to see some of this stuff. What are some of the challenges that are coming down the road?
00:40:14 - 00:41:09
Matt
Yeah. I mean there are a lot of them, so I won't try to get into all of them. If I stay focused on this particular audience, I think number one, there's a good bit of anxiety right now. That takes a toll on our day-to-day work every day, no matter what. I can appreciate that anxiety, but I think a lot of it comes from the amount of attention that LLMs and Chat GPT are receiving in the news right now and the fear that they're creating that this is an even bigger round of the machines are coming for our jobs kind of mentality. I just don't see evidence, at least not yet, that that's really true right now. This is more of an augmentation, it's more of a support tool. It's enabling, the service desk as an example, to take on a wider scope of work without needing to get bigger or without needing to overinvest in training, at the cost of delivery. I see value there and some benefits there.
00:41:10 - 00:41:56
Matt
I think being afraid of it is natural. That makes a lot of sense to me. I get that downside. That's certainly not the only one. When we look across lots of different deployment scenarios right now, I think, the risk that we've been talking about, up to this point, that's still very valid in this conversation. Even though we might be talking about it from an informed place of knowledge, that's not how everybody is coming at this problem right now. A lot of education still needs to happen, for everyone's benefit, around just how are all these models governed differently and use your data differently and you need ... You, as an IT leader alone, aren't the only one who needs to understand that. You need to make everyone else understand it as well. That's a big and difficult undertaking.
00:41:57 - 00:42:51
Matt
Then, I'd say the other side to it is we don't necessarily talk about how these models are able to be successfully perpetuated once they are deployed. I think a lot of companies that we run into right up front are thinking, well, this is something that I press go and it starts to work and then we don't touch it anymore. Then naturally we're going to just start being able to reduce some headcount because we're deflecting all those tickets to a technology, instead. LLMs create new work within the organization in order to keep them functional and working well. We've alluded to a little bit of that without too much detail already in the conversation, but the long term operations cost around deploying and maintaining LLMs is not trivial. It's not something that I think a lot of buyers are modeling, that they're that they're not aware of, they're not really factoring in.
00:42:52 - 00:43:32
Matt
For some organizations benefit that makes managed services that have AI as a component, a more palatable way to do that because you're not absorbing the risk. You get to put it on someone else. But if you're going to take that on, on your own, it's something that you really have to understand and you really need to be able to model out for your organization who's going to do ... What is this work? Who's going to ultimately do this work? Michael was just mentioning that that is somewhat specialized and you can't just give it to anybody walking in off the street. You need to have a plan for that or your model's going to fall down. As soon as that happens, the rest of the organization is going to lose faith and confidence in it, your adoption is going to go away. Then, what was all that effort for?
00:43:33 - 00:44:22
Tim
Yeah. I think to your point about fear, I think that it's incumbent on leaders to remind their teams to apply critical thinking here, a little bit of cynicism. Why is this fear out there? Why are organizations saying, "Hang on. We've got to put a pause on this." Yes, there are various totally legitimate reasons for that, but there's also, if I'm a provider of these systems and you're worried, I can then stop step in and say, "Hey. We know more about this than anybody. You really need to hire us." There is a economic incentive for folks to bring this and for that matter, for folks like us as well, to make sure that there's fear or uncertainty and doubt out there that people are going to be asking us questions.
00:44:23 - 00:44:38
Tim
The other thing is that there are companies out there that are behind. They want to play in this space, they're not there yet. If I'm one of those companies, I'm going to say, "Well, whoa, slow down, everybody. Slow down a little bit." I'm not going to say let me catch up, but that's really what I'm trying to accomplish.
00:44:39 - 00:44:57
Tim
Michael, I'm curious from your perspective, you're a top analyst on cloud and data center strategy. What are some of the parallels and potential drawbacks of entrusting strategic resources, which these clearly are going to be, to a third party? What are some of the issues that that raise?
00:44:58 - 00:46:03
Michael
Well, I think the first thing we need to take care of it, is as this whole issue that Matt mentioned about the Elon Musk letter and all the signatories. I think there's a lot of confusion about general AI and the sort of narrow AI space that we're in at the moment, which LLM fits into the narrow AI. The system does not understand the response that it provides. We interpret it, but it doesn't have capabilities beyond coming out with some useful data mining, if you like, even if it's a surprising result. There's no human-like intelligence behind the system. We are a long, long way from achieving that. There's a lot of research going on, that's great, but we are not there yet. We need to be clear about that. I think there's a little bit of hysteria.
00:46:04 - 00:47:04
Michael
The news media is also talking about job implications and how people's jobs may be threatened by these systems. I think that really is not the case. What generative AI, LLM does is certainly, for example, in the IT context, it improves code quality. I've looked at the anecdotal feedback that developers have provided in forums. It's generally positive, that if you are observant about the responses that are coming out, it does help you, it does improve quality. That has to be a good thing because software quality is a perennial problem. That's a good thing. Used as an assistant, it's not going to threaten people's job, it'll do the opposite. I think it'll improve the quality of their work.
00:47:05 - 00:47:35
Michael
People need to look at the positive side of that and not listen to the hysteria, because I think there's a lot of confusion around the step change that we've talked about that LLM is providing, versus general AI. We are such a long way from general AI that we just should not confuse the two. The Elon Musk letter unfortunately is confusing the two. Yeah.
00:47:36 - 00:48:24
Michael
I think that organizations should certainly have somebody look into how this technology can help them, whether it's using it internally, if they have a research capability, or whether it's something that they need to acquire, look for enterprise solutions that have these capabilities embedded because it's going to be another one of these technologies that you need to have. I'm sure of it. I think over time we'll get over some of these issues that we are seeing that that's been mentioned. Using this technology properly so it doesn't leak confidential information. We shouldn't be inputting in confidential information into these systems.
00:48:25 - 00:49:11
Michael
Personally, I think that the providers of this technology should have an opt-out so that when you make a query, that query is not used for retraining, because what happens then it's used for retraining and then it pops out in somebody else's question. That's where the leakage happens. I think we need an opt-out clause, and as we discussed previously when we started that this may slow down some of the research, but if it means people have confidence in the solution, then I think it's a positive thing. I think we want people to be positive about this technology and not be afraid of it. Personally, I think the opt-out is the way forward.
00:49:12 - 00:49:24
Tim
Great. Marc, are you sharing any strategic or practical cautions with your clients? I'm not talking only about the technology, but also about how they're thinking and talking about the technology, internally.
00:49:25 - 00:50:18
Marc
It's a little bit the same ground that we were talking about before. There's a paper that I uploaded that requires no general knowledge, it's one step away from a general interest article. It's the Eight Things To Know About LLMs. The eight are bulleted right in the abstract part of it. I'm not going to read through the whole thing, but part of what it does do is give people a grounding, because at this point, if you're sitting down to negotiate with a vendor, you have to have some understanding of where the limits are of what this technology does. You want to understand them, from a business process point of view. That means not knowing anything about the details of what's in one model versus the next, or the implications of how they're trained, that's for technologists.
00:50:19 - 00:50:51
Marc
If you're trying to make a business decision, it's where do these things fail in a way that I can understand? What are the limits? How far can they go? What's the repeatability of these things, and what drives that? If I type in, let's say for the chatbots that [inaudible 00:50:37] doing, whether they're doing tech support or they're part of the core IT staff, how often am I going to get the same answer. Not the same in the sense of identical wording, but that it will solve whatever my problem was.
00:50:52 - 00:51:41
Marc
Those are all things that you want to be able to ask. That paper is very concise and gives you that kind of information. That's what we're passing on to them. The other issues like compliance and legal requirements and the kinds of things that Michael has talked about, we also do. At some point you're going to be sitting down and want to sign a contract, and if there is no real understanding of what you just signed for, then you're much more likely to be either disappointed or taken by surprise in some unpleasant manner. Nobody cares if you get taken by surprise, because something works way better and is more affordable, cheaper, all that kind of thing. Where you're going to get disappointed is, I thought I was buying something that could do X and it turns out it does X over two or X over 10. That's what we're driving towards.
00:51:42 - 00:52:06
Tim
Great. I know I had one more question that I shared with you all in advance, but we're down to about 10 minutes here, eight minutes. I'm going to go ahead and flip over to Q&A, because I know that our audience has a whole lot of questions here. A lot of the questions address actually this ... As I say at the intro to the question, I'm not going to ask, I try to bring these questions back to the people who have to make this stuff work every single day.
00:52:07 - 00:52:48
Tim
Let me go ahead and ask a couple questions from our audience. Roy Atkinson asked, "Can you address the concern about career-pathing for support staff in light of the undoubtedly expanding role of AI powered tools?" Before you answer that, I'm also going to add in here a question about to what extent do you think the skill of prompt engineering will be important, as conversational design was for chatbots and prior gen virtual agents? Why don't we go ahead in our original order, which was, let's see ... Sorry. What the heck? Let's just throw it out there. Marc, why don't you start us off just in terms of these.
00:52:49 - 00:53:54
Marc
Sure. Let's go with the career path. Unless you're convinced that artificial general intelligence is here within the next five to 10 years, then as far as being concerned about your job going away or being replaced by a machine that's off the table. What you want to do is ask the question: if I'm augmented, I'm not doing the screwdriver work or the intellectual equivalent of that anymore, where can I move up in the knowledge chain? Years back, I think maybe it's 20 years at this point, there was a book published called The Cognitive Surplus. It was all about organizations have large numbers of people who know the exact same thing and it's not really very beneficial, either to the individuals, or the organization to have a thousand people that can do the same thing. What's beneficial is to somehow shift that to an automated system of some kind and then allow those thousand people, or maybe 10 of them still maintain that information, whatever you're comfortable with, but the balance of them acquire some skill that's more beneficial both to themselves and the organization.
00:53:55 - 00:54:36
Marc
That's at the core of the question that Roy is asking. It's find those things, stop trying to ... Definitely don't compete with the machines, because there'll be incremental advances that continue to do the low hanging fruit. If what you want to do is answer the same question in the same way, every day, that's definitely going to go away. What's necessary is to find out where are the machines doing badly. You're actually seeing the responses, you're in some way an intermediary or maybe you are directly on the receiving end, if it's IT staff doing direct IT support, not for the end users. Where are the machine's screwing up, because wherever they're screwing up, that's where you want to go. That's what they're not good at.
00:54:37 - 00:55:10
Tim
Right. That makes sense. Matt, I'm going to throw this one to you. What is the future of knowledge-centered support? Is there still a need or will there still be a need to curate knowledge articles going forward, or can text from cases be sufficient to help find valid solutions within a chatbot? I think that that was kind of a missed thought there. What do you think, KCS, is there an impact on KCS strategies and frameworks?
00:55:11 - 00:56:19
Matt
Yeah. I think there will be. I don't think KCS is going to go away. When we think of how these types of solutions are deployed, especially in an IT service support context [inaudible 00:55:25] knowledge bases all the time, but that does not mean that they are ready to be consumed by a technology like this. I think there's always going to be curation that needs to happen there. We will likely hit a point in the future where the KCS is supplemented through the LLM, from input from tickets and diagnosis and outcome, but that's not a today or a tomorrow thing. That's the next [inaudible 00:55:55]. For right now, what we've seen, one of the biggest challenges for an organization getting a solution out the door, successfully, is actually redesigning your knowledge base to be consumable by that technology. They do need direction and guidance and how they're structured drives their success.
00:56:20 - 00:56:29
Marc
Absolutely. That's something we didn't talk about, but yes, structuring the internal information to be consumable by these, is a major task. Yeah.
00:56:30 - 00:57:03
Tim
Right. As always, data hygiene is going to be probably first step one to successful implementation of a tool like this. Michael, Open AI is clear in statements that Chat GPT makes mistakes, it's not ready for mission-critical work. How does that square with the commercialization that we're seeing? We talked a little bit about that. Where do we stand with commercialization? Where do we stand with others sort of trying to slow things down. How does that square with the efforts of Open AI, in particular, in terms of is Chat GPT open for business?
00:57:04 - 00:58:13
Michael
Yeah. This is my personal take on it, of course, but my sense is that Open AI sees a path to improving the performance of Chat GPT and the GPT models underlying it. Hence, they clearly want to take up the commercial opportunities. I think also that when they work with partners who are applying this technology in very specific cases, as we've talked about in our discussions, that when you're in a narrower domain, you can certainly tighten the quality of the responses. It's a lot easier to do that. I think that they see that as the way forward to monetize their excellent research. Yeah. That's how I square it basically. They're very careful about the open systems and the fact that they do make mistakes, yeah, but I think that once they work with partners, they will be ... There's going to be general improvement in time, for sure.
00:58:14 - 00:58:20
Tim
Great. Let's see. Matt, you've gone black. I'm not sure if you can still hear us. Are you there?
00:58:21 - 00:58:22
Michael
I think he's got [unintelligible] 00:58:22] problems.
00:58:22 - 00:58:23
Matt
I can. I'm not sure.
00:58:24 - 00:58:35
Tim
Okay. Yeah. Your voice is kind of getting echoey and your screen is blank. There's some connection problems, I think, on your end. I'm going to go ahead and put this last question to [inaudible 00:58:36].
00:58:36 - 00:58:38
Matt
I did lose my primary internet.
00:58:39 - 00:59:06
Tim
Okay. Yeah, Matt, I apologize. I was hoping to have you take us out as the sponsor representative, but we're really having trouble hearing you. Instead of that, I'm going to go ahead and close us out here. We do have some other questions that were asked, that we weren't able to get to. I apologize for that, but we'll do our best to answer those offline if you want to ship those to me in-person. We do record all of these, so let me know if you'd like some answers to those.
00:59:07 - 00:59:45
Tim
This has been absolutely fascinating conversation. Thank you guys very, very much. I'm a little bit upset that we didn't get to the money question, which is about how do you get the investments paid, but we'll regroup. We'll try to have this conversation again and get down into some of those nitty gritties again about training, about how do you talk to the CFO about these issues, what are ... I think tech is going to be asked a lot of questions by folks in other departments across the organization. That communication and that education piece is going to be huge.
[Presentation slide 3. Black background with image with four speakers appearing on the top left of screen. White box with the text “Thank you for attending”. Below this text reads “Please visit our sponsor and view any of the resources featured in the resource section of the attendee console.” Blue CAI logo appears in middle of the screen. Multi-color HDI logo appears on bottom right of the screen.]
00:59:46 - 01:00:28
Tim
That is all we have time for today. Thank you to everyone who joined us for today's webinar. As a reminder, to our audience, you'll receive a follow-up email that will include a link to review a recording of today's presentation. If you found this event useful, please share that with your colleagues. Also, as you leave the event, a short survey is going to pop up in your browser window. We really appreciate it, if you'd share your feedback on this webinar, it helps us tailor our programs to your interests. There were a couple questions about the materials that were mentioned, please take a look at the resources widget, click on that and download anything you find useful. Matt, I see you're back. I'm going to thank you personally. Thanks, CAI. Marc, Michael, it was a fantastic conversation. I look forward to continuing it.
01:00:29 - 01:00:39
Michael
Thanks a lot. Bye, everyone.
[Closing slide 1. Blue CAI company logo with tagline “We power the possible” appears in middle of screen. Company website www.cai.io appears at the bottom center of the screen]
[{"asset_name":"www.cai.io","asset_url":"https:\/\/www.cai.io","asset_type_code":"site"},{"asset_name":"Resources","asset_url":"https:\/\/www.cai.io\/resources","asset_type_code":"page_redirect"},{"asset_name":"Events and Webinars","asset_url":"https:\/\/www.cai.io\/resources\/events-and-webinars","asset_type_code":"page_standard"},{"asset_name":"Generative AI and large language models in IT service and support","asset_url":"https:\/\/www.cai.io\/resources\/events-and-webinars\/generative-ai-and-large-language-models-in-it-service-and-support","asset_type_code":"calendar_event_single"}]
Where are we today in thinking about applying these powerful tools to improve IT service and support?
What are some widely shared misconceptions about LLMs that IT leaders need to avoid?
What’s needed to prepare for and successfully execute a large language model implementation that will help your teams deliver superior IT support and service management?
How can IT leaders convince financial decision-makers to invest appropriately, and continue investing, in LLM?
If you’re interested in learning more about our Intelligent Automation or Service Desk practice, contact us today.
