Artificial intelligence and machine learning are rapidly gaining popularity in every aspect of our daily lives, and cardiovascular medicine is no exception. Here, we provide physicians with an overview of the past, present, and future of artificial intelligence applications in cardiovascular medicine. We describe essential and powerful examples of machine-learning applications in industry and elsewhere. Finally, we discuss the latest technologic advances, as well as the benefits and limitations of artificial intelligence and machine learning in cardiovascular medicine.
Artificial intelligence (AI) is a field of computer science in which computers perform tasks that normally require human thought processes and intelligence. More broadly, AI is a field of engineering focused on building devices that can sense, process, learn, and perform tasks such as perception, recognition, decision-making, classification, detection, and estimation. In other words, AI enables computers to mimic human behavior.1
The rapid increase in computational power and data storage capacity has stimulated the expansion of AI and machine learning in many industrial and scientific areas.2 Because of their potential to improve disease diagnosis, research, and patient care, AI and machine learning have attracted the attention of physicians in the field of cardiovascular medicine (CVM).3 Here, we provide a brief overview of the history and essential features of AI and highlight the latest advances and future applications of AI in CVM.
A Brief History of Artificial Intelligence
The history of AI began in 1943 with the creation, by Warren McCulloch and Walter Pitts, of a computational model for neural networks based on algorithms called threshold logic (Fig. 1).1 This led to further work on neural networks and their link to mathematical computational models called finite automata, or finite-state machines. In the 1950s, Frank Rosenblatt advanced machine learning and described the first machine “capable of having an original idea.”4 In 1958, Rosenblatt wrote, “[We] are about to witness the birth of . . . a machine capable of perceiving, recognizing and identifying its surroundings without any human training or control.”4 Unfortunately, computer technology advanced slowly, and several decades passed before the achievements of early AI investigators were applied practically. Since the historic landing of Apollo 11 on the moon in 1969, computer technology has evolved to the point that even simple devices today are more powerful—in terms of raw processing power—than Apollo's guidance computer was. Today's smartphone is one million times faster than NASA computers were in 1960.2,5
Citation: Texas Heart Institute Journal 49, 2; 10.14503/THIJ-20-7527
Advances in computer technology became possible only after parallel improvements and advances in integrated circuits, sensor designs, nanoscale devices, signal processing, data processing, and data control systems.2,5 To achieve these goals, science and engineering experts had to work together to resolve multiple technologic problems. Progress had to be made in many supporting fields to define problems, formulate solutions mathematically, test them experimentally, and, finally, implement them in practice.6
Supercomputers
The IBM Watson supercomputer combines AI and sophisticated analytic software for optimal performance as a “question-answering” machine. Designed to process data at the rate of 80 teraflops (trillion floating-point operations per second), Watson retrieves information through a combination of natural language processing, image recognition, tone analysis, cognitive computing technology, text mining, and deep machine learning.5 Watson accesses 90 servers that together store more than 200 million pages of information as well as the Watson avatar. Watson uses IBM's DeepQA software, which is designed for information retrieval, language processing, text mining, image recognition, and applies principles of deep machine learning, including cognitive computing technology.5
Supercomputer Artificial Intelligence Algorithms
Supercomputer AI algorithms include the application of common statistical analysis algorithms that can be used not only to improve performance in response to a large amount of data, but also to create models for autonomous predictions. These common algorithms include naïve Bayes (for classifying data probabilistically), decision trees (for predicting labels), random forest, logistic regression, linear regression, support vectors (for separating data into different classes), and k-nearest neighbors (for grouping data) (Table I).7
Machine Learning
Machine learning is the application of statistical analysis algorithms to improve the analysis of a large amount of training data.2 This is done by creating models for autonomous predictions and improving the performance of the analysis algorithm with acquired experience. Machine learning can be divided into supervised, unsupervised, and reinforced (Table I). In supervised learning, data are labeled before training. This expensive and labor-intensive task requires a large amount of reliable data. In unsupervised learning, unlabeled data are exposed to a specific algorithm. In reinforced learning, which is comparatively more powerful, a machine learns through trial and error how to interact with its environment, to generate labels that will organize the data in a manner that achieves the best results.
There are 4 main types of machine learning: reactive, limited-memory, theory-of-mind (able to adjust behavior), and self-awareness (able to have and understand consciousness).5–7Reactive machines cannot form memories or use past experiences to make present decisions (for example, IBM's Deep Blue). They can, however, identify data formats, make predictions, and choose the most optimal possibility among those given. Limited-memory machines can extract and store knowledge from previously learned data. Unlike reactive machines, they can learn from the past by analyzing data to make better predictions. This type of AI is used by virtual voice assistants, self-driving cars, and other technologies. With limited memory, machine-learning architecture increases in complexity. Theory-of-mind machines can understand human behavior and adjust their behavior accordingly. Self-aware machines, which have yet to be created, will have consciousness and be able to form ideas about themselves.1,3,5
Machine learning involves the use of training data to extract relevant features, which are then used as test data to create a model or algorithm. The algorithm, in turn, is used to categorize an object in question into an image and to make a prediction (Fig. 2). An integral part of machine learning is deep learning (Fig. 3).1,3,5,8
Citation: Texas Heart Institute Journal 49, 2; 10.14503/THIJ-20-7527
Citation: Texas Heart Institute Journal 49, 2; 10.14503/THIJ-20-7527
Deep Learning
Deep learning is a subset of machine learning in which a machine uses algorithms and software to train itself to perform tasks (Fig. 4).1,5,6 Deep-learning machines apply the concept of neural networks to a large amount of data. In theory, a neural network consists of an input layer, one or more hidden layers, and an output layer. Every hidden layer, in turn, consists of several processing units that functionally mimic human neurons (Fig. 2). Single-vector inputs are transformed by subjecting them to the multiple hidden layers. Thus, deep learning is characterized by the analysis of data from hierarchical layers that functionally mimic neural networks to make more complex predictions. In complex networks, the multiple hidden layers mimic the neural network of the human brain; each neuron is connected to every neuron in the previous hidden layer. The last hidden layer is connected to the output layer that shows the network decision and prediction. Current examples of deep learning are photo recognition (Yahoo and Flickr), speech recognition (Apple and Amazon), and self-driving cars (Google, Intel, and Tesla).
Citation: Texas Heart Institute Journal 49, 2; 10.14503/THIJ-20-7527
Powerful Examples of Artificial Intelligence in Use Today
The aviation industry was one of the first fields to embrace and implement AI. By incorporating AI-based simulation into pilot training, the industry was able to improve flight skills, reduce pilot error, and reduce the number of flight accidents by more than 90%.2,3,5,6 In actual flight, multiple redundancies and back-up mechanisms (for example, the presence of a copilot in the cockpit, equipment and procedural redundancy, and routine use of a checklist before takeoff ) were augmented by AI-based autopilot programs. Autopilot enabled a computer to fly the airplane by using a series of sensors and executing algorithms usually programmed by the pilot. In the early days of flight automation, pilots were still needed to monitor wind and weather conditions, track fuel consumption, and take over flight control during air turbulence or other concerning situations. However, recent advances in AI have made an autopilot capable of doing most things that a human pilot can do. During flight, the automated flight-management system typically determines the most efficient way to execute flight variables without pilot input. Many leading airplane manufacturers are now producing and testing pilot-free, robot-controlled planes.
Other powerful examples of AI in use today include self-driving cars built and tested by Google, Intel, and Tesla; flying drones and speech-recognition programs made by Apple and Amazon; and photo-recognition programs made by Yahoo and Flickr. All of these AI applications use deep-learning algorithms inspired by human neural networks and transient memory. In the foreseeable future, most—if not all—vehicles may be driverless, with the anticipated benefits of safer and more economical means of transporting people and goods.6,9
Applications in Cardiovascular Medicine
The main purpose of AI in CVM is to improve data collection and analysis, in order to diagnose disease more accurately, detect disease earlier, and better predict outcomes (Table II).3,10 The ultimate goal is better quality of care, more appropriate disease surveillance, and more optimal timing of interventions.3 Applying improved AI algorithms to advanced imaging techniques and technologies will improve patient care. Artificial intelligence will also help uncover novel associations between data and disease. The hope is that the use of AI will reduce human error, make medical care less costly, and facilitate professional data sharing in a variety of formats. In addition, the use of supercomputers and various statistical, algebraic, and logical algorithms will enable more accurate analysis of large datasets. For example, Huang and colleagues11 used machine-learning techniques to show that an AI-based computational probability value algorithm could predict impending acute kidney injury after percutaneous coronary intervention 1 to 2 days before it could be diagnosed through routine clinical testing. Yet despite AI's rapidly growing and evolving role in CVM, its practical implementation remains problematic (Table III).3,10,12 Challenges include integrating data into the clinical workflow; interpreting and understanding machine-learning models; identifying AI-based misdiagnoses; and addressing and resolving social, legal, and regulatory issues that may arise.
Applications in Cardiovascular Imaging
The usefulness of AI in cardiovascular imaging is widely accepted.3,10,13 One existing application is a computer system that autonomously analyzes and interprets retinal images for signs of referable diabetic retinopathy.14 This application, one of the first autonomous AI diagnostic systems able to generate a screening result with no clinician interpretation, has reportedly high sensitivity, specificity, and success in providing adequate imaging.13
Several cloud-based, searchable repositories are available for evaluating AI-based image analysis systems. A prime example is the Stanford Medical ImageNet, which contains more than 230,000 chest radiographs from patients with various clinical conditions.15 (A recent analysis of this repository by an AI-based deep-learning neural network algorithm was sensitive and specific enough to produce results equivalent to expert interpretations.15) Other cloud-based imaging repositories include the Radiological Society of North America (RSNA) database of chest radiographs, the RSNA database of more than 25,000 computed tomographic scans of the brain, the EchoNet database of more than 10,000 echocardiograms, the Lower Extremity Radiographs (LERA) database of lower extremity radiographs, and the MRNet database of magnetic resonance imaging (MRI) scans.13,15,16
Applications in Ultrasonographic and Echocardiographic Imaging
Human sonographers and echocardiographers face several challenges in imaging, including unnatural hand-eye coordination, interpretation of unintuitive images by the untrained eye, and nuances of imaging patients with different body habitus or medical conditions. Conversely, AI-based programs can overcome these challenges while improving the efficiency of clinical decision-making and reducing the costs of evaluation and treatment. Such programs also empower primary care providers to diagnose various medical conditions earlier and perform cardiac monitoring and triage more efficiently and productively.
The application of AI to echocardiographic imaging is rapidly increasing.17 Many imaging companies are developing AI-assisted technologies that provide images more accessible to and more easily interpreted by expert and nonexpert users alike, provide real-time guidance and automated quality feedback, and enable accurate interpretation.
Examples of AI-based automated ultrasonographic systems include the GE Healthcare Venue for duplex imaging of the inferior vena cava; the DiA LVivo for assessing left ventricular ejection fraction on a GE Healthcare Vscan system; and the Tomtec AutoStrain for assessing peak systolic longitudinal strain on a Philips Epiq recording system (Fig. 5).18–20 An AI-based echocardiographic imaging system (Caption Guidance; Caption Health, Inc.) emulates the guidance of an expert sonographer by providing more than 90 types of real-time feedback and instructions.20 The system's on-screen “quality meter” shows in real time how close users are to capturing a diagnostic-quality image. Its “Auto-Capture” algorithm automatically records diagnostic-quality, hands-free images. In addition, the program automatically generates a left ventricular ejection fraction measurement from a single view or from multiple views (Fig. 6).20 The advent of AI-assisted automated, real-time quantification of transesophageal echocardiograms is of particular value for volume analysis, real-time 3-dimensional (3D) Doppler measurement, and 3D valve analysis during valvular or surgical procedures.17,20–22
Citation: Texas Heart Institute Journal 49, 2; 10.14503/THIJ-20-7527
Citation: Texas Heart Institute Journal 49, 2; 10.14503/THIJ-20-7527
Applications in Other Types of Imaging
Advances in AI and technology have led to reduced radiation doses, integrated imaging modalities, and improved image processing. Together, these advances have facilitated better, faster, safer, and more cost-effective care. An AI-guided system for acquiring cardiac magnetic resonance (CMR) images (so-called “one-click MRI”) (HeartVista, Inc.) has reduced scanning time from 90 minutes to 15 minutes. An AI-based algorithm for calculating fractional flow reserve from coronary computed tomographic angiograms (HeartFlow FFRCT) can diagnose the severity of coronary artery disease with very high sensitivity and specificity (Fig. 7).21,23
Citation: Texas Heart Institute Journal 49, 2; 10.14503/THIJ-20-7527
In radiology, deep-learning algorithms and avatar-supported virtual reality are used by several radiologic equipment companies to improve patient care by implementing landmark detection, range detection, range adaptation over time, isocenter positioning, and patient direction analysis.19
Another major AI application is fusion imaging, including 3D image fusion for case planning during interventional procedures.24 Such imaging involves the use of preoperative computed tomographic angiograms for procedural planning, parallax correction, and automated C-arm positioning to determine vessel origin. This technologic improvement eliminates errors and offers virtual procedural planning. More recently, Philips has introduced Fiber Optic RealShape (FORS) imaging technology that can be used during endovascular abdominal aortic aneurysm repair to simplify catheter deployment in patients with a challenging anatomy, thus enabling more efficient cannulation of the endo-graft's contralateral gate (Fig. 8).25,26 This technology is currently being evaluated in the European Union and will be evaluated soon in the United States. Philips is also partnering with Microsoft to introduce holographic imaging for endovascular interventions. The combined technology, which integrates the Philips Azurion radiographic imaging platform with Microsoft HoloLens 2 smartglasses, provides the operator easier and more effective access to imaging controls and to interventional equipment (Fig. 9).18,25,26
Citation: Texas Heart Institute Journal 49, 2; 10.14503/THIJ-20-7527
Citation: Texas Heart Institute Journal 49, 2; 10.14503/THIJ-20-7527
In the near future, advances in AI imaging in the areas of virtual reality, augmented reality, mixed reality, extended reality, simulation, and holographic imaging will substantially improve interpretation, decision-making, and education in CVM.24–26 To understand the future role of these applications in medicine, it is important to understand the concept behind each.
Virtual Reality
In simple terms, virtual reality is a complete-immersion experience that shuts out the physical world. Using virtual reality devices such as Oculus Rift or Google Cardboard, users are transported into real-world and imagined environments such as a cardiac catheterization laboratory or an interventional endovascular suite. In virtual reality, an interventionalist can interact with a computer-simulated 3D environment through special electronic equipment (for example, a visored helmet or sensor-equipped gloves). Some physicians already use virtual reality systems to plan procedures or educate patients about various treatments. Using a virtual reality headset, the viewer can clearly see every step of the interventional procedure, and an instructor—who appears as an avatar—can guide the viewer in the 3D space.18,24–26
Augmented and Mixed Reality
Augmented reality adds digital components to a live view, typically through a camera. Mixed reality combines elements of both augmented and virtual reality, where real-world and digital objects interact. Mixed reality technology has recently been applied interventionally with use of the Microsoft HoloLens, one of the most advanced mixed-reality technologies.18,24 Holographic imaging and virtual reality have been useful during diagnostic and endovascular procedures. One example is use of the previously mentioned system integrating the Philips Azurion radiographic imaging platform and Microsoft HoloLens 2 smartglasses.
Extended Reality
Extended reality broadly incorporates various sense-enhancing technologies that provide additional information about the actual world or create completely unreal, simulated worlds to experience. All of the reality tools discussed so far will in time become more important, integral, and essential to simulation applications in the field of CVM.
Holographic Imaging
Holography is the process of creating 3D images (“holograms”) with use of laser beams, the properties of interference and diffraction, light intensity recording, and illumination (Fig. 10). The holographic process was discovered in 1948 by Dennis Gabor while he was performing research to improve electron microscopy.26–28 In brief, a light field from a light source scattered off of objects is recorded and reconstructed when the original light field and original object are no longer present. Digital holography is a 2-step process. First, a radiographic image is recorded and then transformed into a photographic record. Second, the image is reconstructed as a hologram that is transformed into a virtual image. This imaging process involves interference between 2 holograms that can change according to the position of the viewer. There are 2 types of holography: conventional and dynamic. Conventional holography produces a static image of a 3D object. Dynamic holography adds motion to the 2- or 3-dimensional object. Holograms currently in use are autostereoscopic and enable objects to be seen in 3 dimensions without the use of 3D glasses.
Citation: Texas Heart Institute Journal 49, 2; 10.14503/THIJ-20-7527
Future Applications of Holography in Radiology and Cardiovascular Medicine
Any 3D medical imaging dataset—for example, those used in computed tomography, echocardiography, chest radiography, angiography, and MRI—can be converted into a digital hologram.26,28 This is achieved with use of computer modeling software and specific proprietary algorithms. Holography can be used to teach anatomy to physicians-in-training in an appealing and intuitive way. It can be used by physicians to provide a visual image when explaining anatomy or various procedures to their patients. In addition, holography can be used to store and retrieve imaging data and so will likely become the storage medium of choice in the future. In contrast with data stored on standard optical disks, holographic data can be stored as a 3D volumetric density that requires physical storage space 1,000-fold smaller than a DVD or optical disk.26,28 Holographic data can also be read at a rate exponentially faster than that afforded by currently available technologies.
Meanwhile, the use of holography in radiology and CVM continues to evolve. This imaging technique will be particularly useful for interventionists and surgeons during preprocedural planning, surgery, or endovascular interventions. At present, holographic technology for use in radiologic imaging lacks sufficient spatial resolution. However, holographic techniques are already benefiting physicians through their application in echocardiography and in guidance during interventional procedures. Recent advances in holographic 3D image displays in free space give physicians the ability to reach for a tool and manipulate it with ease and precision.25,26
Current Challenges in Implementing Artificial Intelligence in Cardiovascular Medicine
A major challenge in implementing AI in CVM is determining how best to integrate the available data into the clinical workflow (Table II). Machine learning models are only as good as the training data entered.2,6,21 We must learn not only how to create machine learning models, but also how to understand and interpret them3 and identify an AI-based misdiagnosis when a problem occurs. It is important to determine the acceptable threshold for AI-induced errors and what to do if those mistakes are ultimately fatal. Several substantial regulatory, social, legal, and ethical issues remain to be addressed. From a regulatory standpoint, the United States Food and Drug Administration (FDA) currently considers AI to be a medical device that requires proof of safety and efficacy before receiving final approval. This gives the FDA the opportunity to monitor and regulate such technology from premarket approval to aftermarket performance.29 Currently, the algorithms used in AI offer limited insight into how the models based on them are derived.3,6,10,13
Photograph shows Frank Rosenblatt with the first machine capable of perceiving, recognizing, and identifying its surroundings without any human training or control.
Photograph by Sol Goldberg. Courtesy of Division of Rare and Manuscript Collections, Cornell University Library. Used with permission.
Diagram shows an example of pattern recognition in machine learning through use of training data, feature extraction, model creation, model recognition, and prediction.
Diagram shows a neural network–like algorithm consisting of an input layer, hidden layers, and an output layer. This is the basis of deep learning.
Courtesy of Mayo Foundation for Medical Education and Research.8 All rights reserved. Used with permission.
Diagram illustrates the interactions between artificial intelligence, machine learning,and deep learning.
Image shows peak systolic longitudinal strain (%) measured with use of the Tomtec AutoStrain application on Philips Epiq echocardiographic equipment.
Image courtesy of Philips.18 Used with permission.
Image shows the on-screen output of an artificial intelligence–based echocardiographic imaging system (Caption Health, Inc.), including a quality meter, autocapture images, and automated left ventricular ejection fraction measurements in multiple views.
Image courtesy of Caption Health, Inc.
Graph compares the diagnostic accuracy (sensitivity and specificity) of fractional flow reserve calculation with use of computed tomographic angiography (CTA), artificial intelligence-guided coronary CTA (HeartFlow FFRCT ), positron emission tomography (PET), and single-photon emission computed tomography (SPECT).
P values are for comparisons of area under the curve for CTA, PET, and SPECT versus HeartFlow FFRCT .
Adapted with permission from Driessen RS, Danad I, Stuijfzand WJ, Raijmakers PG, Schumacher SP, van Diemen PA, et al. Comparison of coronary computed tomography angiography, fractional flow reserve, and perfusion imaging for ischemia diagnosis. J Am Coll Cardiol 2019;73(2):161–73.23
Three-dimensional (3D) computed tomographic angiograms, obtained with use of the Philips Fiber Optic RealShape system in biplane viewing mode, shows catheterization of a tortuous iliac artery in A) anteroposterior and B) caudocranial views. Such images provide useful 3D information for device manipulation during interventional procedures.
Reproduced from van Herwaarden JA, Jansen MM, Vonken EJPA, Bloemert-Tuin T, Bullens RWM, de Borst GJ, Hazenberg CEVB. First in human clinical feasibility study of endovascular navigation with Fiber Optic RealShape (FORS) technology. Eur J Vasc Endovasc Surg 2021;61(2):317–25.25
Photograph illustrates the concept of augmented reality for endovascular intervention through holographic imaging with use of the Philips Azurion imaging platform and Microsoft HoloLens 2 smartglasses.
Image courtesy of Philips.18 Used with permission.
Graphic image shows the mechanism of holographic imaging.
Contributor Notes
